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This document presents key findings about the team 
evaluation process used in seven workplace basic skills programs in 
1993 and 1994. An introduction summarizes the methodology used to 
develop the case studies and key findings about the team evaluation 
process. These key findings present the strengths and challenges of 
using evaluation teams, along with recommendations for those 
interested in trying this approach. Recommendations include the 
following: evaluation teams need to create time, interest, expertise, 
a supportive context, a code of confidentiality, and an ethic of 
continuous improvement; teams need to customize the process to fit it 
to the site's unique needs and opportunities; teams should clarify 
evaluation's role in the larger program planning process; teams 
should develop evaluation procedures consonant with the company's 
culture; teams should use familiar and accessible resources; and 
teams need to be realistic. Six case studies describe how the team 
evaluation process was used in seven workplace education programs: 
Bull Worldwide Information Systems; Datatec Industries; Haartz 
Corporation; Norton Company; Project REACH; and Victoria General 
Hospital. Each case study follows this format: program background 
(history, goals, and description); background on the team (history 
and stakeholders represented); the process which the team went 
through; ref]ections and various appendixes, such as sample 
data-gathering instruments or reports generated by the teams* 
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EXECUTIVE SUMMARY 



TEAM EVALUATION : 
CASE STUDIES FROM 
SEVEN WORKPLACE EDUCATION PROGRAMS 

In 1993 and 1994, workplace education resarchers Laura 
Sperazi and Paul Jurmo field-tested a team-based evaluation 
methodology in seven workplace education programs. This "team" 
approach involves key program stakeholders in a process of (1) 
building an education team; (2) clarifying what information the team 
would like to generate in an evaluation and designing a strategy and 
tools for collecting that information; (3) collecting, organizing, 
analyzing, and reporting findings; and (4) taking necessary follow-up 
action. 

The researchers had developed this "team" or "collaborative" 
approach two years earlier in evaluations of workplace education 
programs sponsored by the Massachusetts Workplace Education 
Initiative and other sources. This team model was conceived as an 
alternative to more-familiar evaluation scenarios in which (1) 
program staff carry out informal monitoring of program operations 
and/or (2) more-formal evaluations are carried out by outside 
evaluation specialists. 

In this project, the researchers wanted to explore the premise 
that evaluation carried out by a team of internal "stakeholders" could 
borrow the best from the existing approaches to evaluation while 
avoiding their limitations. More specifically, a well-planned 
evaluation conducted by a team of internal stakeholders could 
produce information meaningful to those stakeholders. It would help 
them understand what the program might achieve and was 
achieving, and identify actions which they and others might take to 
strengthen the program. The researchers felt that this evaluation 
model also had a natural "fit" with workplaces promoting team 
decision-making and continuous improvement, an approach to 
workplace organization which the researchers wanted to support. 

This document presents case studies which describe how the 
evaluation methodology played out in the seven sites, along with 

iii 
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lessons which t±ie researchers and team members learned about the 
evaluation process. These "lessons" are summarized below. 

Strengths of the team evaluation model 

a. Team evaluation can ties in with team management model. It 
"fits" with workplaces shifting toward collaborative decision- 
making and continuous improvement. 

b. Team evaluation can get all stakeholders involved so the 
evaluation and the education program focus on meaningful 
outcomes. 

c. Evaluation standards are developed internally, resulting in 
reasonable, meaningful focal points for the evaluation and the 
program. 

d. With all stakeholders involved, evaluation can become another 
collectively-valued "essential program component" for which 
resources must be allocated. 

e. Team evaluation can provide opportunities for staff 
development ~ for all team members but especially for 
program participants. 

f. Team evaluation can produce evaluation procedures which can 

be replicated by the program itself and by others. 

g. Team evaluation can provide an "audience" for stakeholders' 
ideas. 

Challenges of the team evaluation model 

a. Team evaluation takes time. 

b. Team evaluation requires a mix of special skills. 

c. Team evaluation requires particular attitudes and values. 

d. Team evaluation requires discipline. 

e. If not properly done, the team evaluation process can be 
skewed to serve the interests of just one or a few stakeholders. 

f. If confidentiality is not respected, team evaluation can 

needlessly jeopardize the security of those involved. 

g. Team evaluation requires special knowledge and procedures to 
minimize bias and ensure validity and reliability. 

h. It is difficult to introduce a "team" approach to evaluation in a 
program or workplace not already "team-oriented." 

3. Recommendations for those interested in trying team 
evaluation 

a. Evaluation teams need to have or create: 

iv 
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• Time 

• Interest (motivation, "team values") 

• Expertise 

• A supportive context 

• A "code of confidentiality" 

• An ethic of continuous improvement 

b. To ensure that teams have the above elements in place, those 
organizing teams should use the following team-building 
activities: 

• Select members carefully. 

• Negotiate roles. 

• Stress the need for continuity. 

• Provide extra training and technical assistance. 

• Organize team activities to allow aU members to participate 
actively. 

• Set a "climate" of respect. 

c. Teams need to customize, refine, streamline the process to fit it 
to the unique needs and opportunities of each site. 

d. Teams should clarify where evaluation fits into the larger 
program planning process. 

e. Teams should consider calling on the expertise of an outsider if 
necessary. 

f. Teams should "get their feet wet" quickly, not waiting too long 

to start. 

g. Teams should understand that their goals for the program will 
change as they learn more about it. 

h. Teams should develop evaluation procedures consonant with 
the culture of the company and education program. 

i. Teams should use resources which are familiar and accessible, 
j. Teams should balance "product" (reports, etc.) with "process" 

(the dialogue created among stakeholders), 
k. Teams need to be realistic and fair in what they expect from an 

education program. 
1. Teams should beware of intra-organizational politics, 
m. Funders and other "higher-ups" should show interest in the 

team's activities without interfering in them, 
n. Teams should make recommendations wliich are realistic. 



Primary funding for this project was provided by the National 
Institute for Literacy in Washington, D.C. Additional support for the 
Canadian field-site was provided by ABC CANADA, a national 
workplace education service based in Toronto. 

v 



9 



INTRODUCTION 



Background on t±Lis document 

This introductory document presents key findings about the 
team evaluation process used in seven workplace basic skills 
programs in 1993 and 1994. Accompanying this document are case 
studies which chronicle the process of conducting team-based 
evaluations from the first stages of selecting and preparing a team to 
the final stages of reporting findings and using results to improve the 
program. 

When we began this project, we had good working ideas about 
how to implement a team-based evaluation in a workplace basic 
skills program. We had piloted an early version of the method with 
six programs which were part of the Massachusetts Workplace 
Education Initiative in 1990-91. We had been so encouraged by the 
results that we continued to evolve the method with other programs 
in 1992. 

We saw the use of evaluation teams as a promising alternative 
to more-famHiar evaluation scenarios in which ( 1) evaluation is done 
internally by program staff on an ad hoc, informal basis, or (2) 
control for evaluation is put in the hands of an outside evaluation 
"expert." We felt that evaluation carried out by a well-organized 
team of internal "stakeholders" could borrow the best from the 
existing approaches to evaluation while avoiding their limitations. 
More specifically, a well-planned evaluation conducted by a team of 
internal stakeholders could build stakeholder understanding of and 
ownership for the program. We felt that this evaluation model also 
had a natural "fit" with workplaces promoting team decision-making 
and continuous improvement, an approach to workplace organization 
which we wanted to support. 

In late 1992, the National Institute for Literacy provided us 
witii funding to further develop this methodology with seven 
programs. This gave us a unique opportimity to intensively explore 
many of the assimiptions and procedures imbedded in our earlier 
model. From late 1992 through early 1994, our early ideas took 
new shape and meaning as we interacted with the many people who 
became members of our evaluation teams, and who were themselves 
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struggling to understand what workplace education programs can 
legitimately be expected to achieve. 

At the end of this challenging process of selecting sites, trying 
out our team evaluation methodology, and analyzing our experience, 
we produced two documents: 

• Team Evaluation: A Guide for Workplace Education 
Pro grams , which was prepared for other workplace education 
teams interested in conducting their own internal evaluations. 

• This document, Team Evaluation: Case Studies from Seven 
Workplace Education Programs , which details the process we 
went through in our field sites, and analyzes that experience. 

This project also produced a third document which drew on 
sources other than our seven field sites: 

Workplace Education: Stakeholders' Expectations. Practitioners' 
Responses, and the Role Evaluation Might Plav . Based on a 
review of the literature and interviews, this report analyzes 
the purposes of workplace education programs; approaches to 
workplace education; the state of the art of evaluating those 
programs; recent attempts to develop a new, "collaborative" 
program model; and steps which stakeholders might take to 
strengthen the field in general and evaluation in particular. 



How this document is organized 

This case study document is divided into seven smaller 
documents: 

This introductory document which summarizes the 

methodology used to develop the case studies and key 
findings about the team evaluation process. 

Six case studies which describe how the team evaluation 
process was used in a total of seven workplace education 
programs. Each case study includes various appendices, 
such as sample data-gathering instruments or reports 
generated by the teams. 
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Next steps 



We feel this project has produced a tremendous amount of new 
knowledge about the value of - and procedures for - involving all 
stakeholders as active players in workplace basic skills efforts. 
Many questions remain unanswered ~ some old ones and some new 
ones which emerged from these 11/2 years of experience. As we 
wind up this project, we look forward to working with others who 
would like to explore these questions further in the pursuit of 
creating "new and improved" approaches to workplace education. 
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THE METHODOLOGY USED 
TO DEVELOP THE CASE STUDIES 



The seven sites 

The two researchers originally intended to select six workplace 
education programs to serve as study sites. These programs would 
have to have been already well-established and have met the 
following criteria; 

1. Each program would be managed by an existing team 
composed of a range of stakeholders, including learners, 
instructors, managers, and union representatives (where a 
union existed). These teams would have to be interested in 
conducting their own evaluation and able to devote time to 
working with the researchers to conduct a pilot evaluation. 
In lieu of such an existing team, programs would have to be 
willing to form one for the purposes of the study. 

2. The workplaces would be moving toward a "high- 
performance" organization model characterized by team 
decision-making and commitment to continuous 
improvement. Such a workplace would preferably also have 
an interest in tying its workplace education program into 
those larger changes in organizational structure. 

Laura Sperazi chose three sites from the Massachusetts 
Workplace Education Initiative. She had previously served as an 
evaluator for various Initiative sites, and ~ with her help -- the 
Initiative had already begun orienting its programs to a "team" 
approach to planning and evaluation. Through her contacts with the 
Initiative network, she identified three programs willing and able to 
participate as field sites and which met the site-selection criteria. 

New Jersey-based Paul Jurmo did not have ready access to the 
same kind of network of workplace education programs available in 
Massachusetts. He used his contacts in the field to discuss the 
feasibiUty of finding tliree sites which met the selection criteria. 
After a time-consuming series of discussions with a dozen programs, 
he was able to find four which met the criteria. (Note that several 
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programs expressed interest in participating, but were not 
sufficiently well-established to take on the additional responsibilities 
of participating in this research project. Also note that, to enable the 
study to include a site in Canada, additional funding for travel, 
telephone, and related costs was provided by ABC CANADA, a 
national technical assistance organization based in Toronto.) 



The seven sites represented a mix of industries, populations 
served, type of instruction, and locations, as profiled below: 





INDUSTRY 


POPULATION 
SERVED 


TYPE OF 
INSTRUCTION 


BULL 

WORLDWIDE 
INFORMA- 
TION 
SYSTEMS 
(Mass.) 


Manufacturer 
of computer 
parts 


Twenty-eight 
"associates" of 
mixed ethnic 
and language 
backgrounds 


• 3 ESL classes 


DATATEC 
INDUSTRIES 
(New Jersey) 


Manufacturer 
of computer 
parts 


One dozen 

Latina 

immigrant 

women 

assembly 

workers 


ESL (in small 
groups 
organized by 
skill level) 


HAARTZ 

CORP. 

(Mass.) 


Manufacturer 
of vinyl fabrics 


Fifty line 
employees, 
mostly white 
men, with a few 
women. 


• 2 math 
classes 

• 2 language 
arts classes 

• GED class 


NORTON 

COMPANY 

(Mass.) 


Manufacturer 
of grinding 
wheels and 
related 
abrasive 
products 


One hundred 
employees, 
including many 
older white 
men who have 
worked for the 
company for 
some time 


• 3 math 
classes 

• 3 language 
classes 

• 2 blueprint 
reading classes 

• 1 introduction 
to 

measurement 
class 

• 1 machine 
maintenance 
class 
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STATE 
UNIVERSITY 
OF N.Y. (at 
Albany, NY) 


Educational 
institution 
(state 
university) 


Custodial 
workers on 
state university 
campus 


10-week basic 
reading and 
writing course, 
using whole 
language 
approach. 


STATE 
UNIVERSITY 
OF N.Y. (at 
Stony Brook, 
NY) 


Educational 
institution 
(state 
university) 


Two dozen 
custodial and 
maintenance 
workers in 
campus 
residence halls 


• Initial 7- 
week summer 
math course. 
•Subsequent 

technical 
communications 
(reading and 
writing) course. 


VICTORIA 
GENERAL 
HOSPITAL 
(Halifax, Nova 
Scotia) 


Major teaching 
hospital for 
Maritime 
Provinces 


Primarily 
Canadian-bom 
hospital 
workers from 
food services, 
custodial, and 
other 

departments 


• Basic reading, 
writing, math 
(for personal 
and job-related 
development) 

• GED prep. 



Researchers' roles 



In five of the seven sites, the researchers served as facilitators 
of a team evaluation process. This process varied somewhat from 
site to site depending on the particular needs, resources, and 
schedules at each site. In general, however, the process involved: 

(1) forming and preparing the teams, 

(2) helping the teams identify what information tliey needed 
and how they would collect it, 

(3) collecting, organizing, analyzing, and reporting findings, and 

(4) taking necessary follow-up actions. 

In tliose five sites, the researchers worked with site 
coordinators to lead the teams through the various required steps. 
The researchers also documented the process and the various 
products (e.g., data-gathering instruments, minutes of team meetings, 
evaluation reports) which emerged. 
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In the remaining two sites (at the two State University of New 
York campuses), Paul Jurmo trained two workplace educators who 
had previously contracted to help those programs with planning- and 
evaluation-related needs. In the training, the consultants were 
introduced to the team evaluation process and figured out how they 
might apply it in their two sites. Paul Jurmo then maintained regular 
long-distance communication with the two consultants to (1) provide 
them with ongoing technical assistance and (2) document the 
evaluation process as it played out in the two sites. 

How the case studies were prepared 

The above interactions with the seven teams varied from six to 
twelve months, depending on the site. Toward the end of those 
interactions, the researchers compiled drafts of case studies 
documenting the process and lessons learned for each site. These 
drafts were submitted to the site teams for feedback to ensure 
accuracy and confidentiality. The researchers then revised the drafts 
accordingly and compiled them in this document. 

As each case study was compiled, the feedback from team 
members and the researchers' personal observations were 
incorporated into the end of each case study. Those analyses by 
team members and by the researchers were then summarized and 
further analyzed into the following "Lessons Learned" section. In it, 
the researchers present what their work with the seven sites has 
demonstrated about the team evaluation process. 
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LESSONS LE A RNED : 
KEY FINDINGS ABOUT TEAM EVALUATION 



This section summarizes what the researchers learned about 
the team evaluation process. These findings draw directly from the 
project's seven test sites, as well as from prior work done by the 
researchers in other workplace education evaluations --"team-based" 
and otherwise. 

This summary presents the strengths and challenges of using 
evaluation teams, along with recommendations for those interested 
in trying this approach. 



1. Strengths 

a. Team evaluation can tie in with team management modgl . 
The team evaluation process can complement and support the 
team approach to management now being adopted by many 
North American workplaces ~ and vice versa. Companies 
interested in developing mechanisms for involving employees 
in collaborative decision-making might see a collaborative 
evaluation of their employee education program as an 
opportunity for showing what workers can do and helping 
them develop team decision-making skills they can use in 
other company contexts. Conversely, a workplace education 
program interested in carrying out a team evaluation will 
likely have a "head start" if the company within which it is 
operating already has a history of "teamwork"-- a collaborative 
ethic, mechanisms, and expertise - which the education 
program can build on. 

b. Team evaluation can get all stakeholders involved so the 
evaluation and the education program focus on meaningful 
outcomes . By involving a more complete range of stakeholders 
in identifying program goals, the team evaluation process 
allows (1) the evaluation to focus on those particular goals and 
thereby clarify for stakeholders whether their expectations are 
being met and (2) the education program to better know how it 
might serve stakeholders' needs in the future. Historically, 
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programs and their evaluations have tended to not take into 
account the expectations of all stakeholders, but to define 
program goals and evaluation criteria more narrowly. Team 
evaluations have generated new interest in workplace 
education programs by (1) providing a system of "checks-and- 
balances" to ensure that all stakeholders are given a chance to 
decide what information to focus on in the evaluation, and (2) 
opening stakeholders up to a broad range of outcomes they 
might not previously have considered. 

c. Evaluation standards are developed internally, resulting in 
reasonable, meaningful focal points for the evaluation and the 
program . The team evaluation process allows stakeholders to 
think critically about what a program can reasonably achieve 
and therefore be held accountable for. Members can, for 
example, get away from expecting the program to by itself 
"improve productivity;" instead, members can define clearer, 
more specific objectives for the program to focus on. This is in 
contrast to situations where program standards are set 
externally ~ or even internally ~ by sources who are either 
unfamiliar with the workplace's actual needs or who haven't 
given much thinking to them. Wlien program objectives are 
not carefully negotiated, the resulting standards are liable to 
push the program down paths which do not reflect 
stakeholders' actual needs and interests. 

d. With all stakeholders involved, evaluation and education can 
become collectively-valued "essential components" for which 
resources must be allocated . Evaluation is too often seen as an 
imposition or afterthought rather than an essential element of 
a solid program. A team evaluation process can demonstrate 
the value of a well-crafted evaluation to all stakeholders. They 
are thus more likely to invest time and other resources in 
evalaution in the future. 

A solid evaluation can also increase the credibility of the 
education program itself, by demonstrating what the program 
is achieving and that a team of stakeholders thinks it is worth 
investing their time in. 

e. Team evaluation can provide opportunities for staff 
development - for all team members but especially for 
program participants . By going through a well-planned team 
evaluation process, all members can develop particular 
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knowledge, values, and skills useful to the organization and 
individuals. For example, members can learn how to define 
and solve a problem, communicate with others within the 
organization, and develop action strategies. They can also 
develop self-confidence and stronger relationships and expand 
their vision of what they and the organization can accomplish. 
Participation in the team evaluation process might thus be seen 
as a valuable staff development activity. It can also be a 
unique life experience for program participants normally not 
expected to t^e on such an active leadership role. 

f. Team evaluation can produce evaluation procedures which 
can be replicated by the program itself and by others . The 
process of collectively designing, using, and fine-tuning 
evaluation procedures is initially time-consuming. However, 
the resulting procedures can be considered as "templates" 
which can be revised and used again by the program without 
requiring all of the same preparations. Other programs with 
similar evaluation needs might borrow from those well-crafted 
procedures, as well, thereby saving them some of the time 
otherwise devoted to the design stage. 

g. Team evaluation can provide an" audience" for stakeholders' 
ideas . The team serves as an "audience" for the ideas which 
team members and others might have been carrying around in 
their heads about the education program and the workplace. 
Teams can thereby call up ~ and build on ~ the considerable 
prior knowledge, positive motivations, and resources which 
stakeholders bring with them. Such a process can encourage 
stakeholders to speak up, get involved, think, and make 
suggestions in ways they normally would not. The team needs 
to be be ready to really listen and respond to those ideas ~ 
stakeholders' "vision" about what a better program and 
workplace might look like. 

2. Challenges 

a. Team evaluation takes time . Rather than rely on a single 
evaluator (whether an external or internal one) to do most of 
the work, a team evaluation requires a number of people to 
commit time to the preparations and other tasks required. The 
process will almost inevitably have false starts, run into 
obstacles, and otherwise require patience and perseverance. 
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However, most workplaces are busy places these days, 
undergoing many changes, and placing many demands on 
employees. It is hard for most stakeholders to pay attention to 
the education program at all ~ let alone committing time to an 
evaluation. This is even more true in small or short-term 
education programs; members are likely to wonder whether 
going through a time-consuming evaluation process is really 
worth it if the program is going to last only six months and 
serve twelve workers. Members lacking time will be reluctant, 
unwilling, or unable to participate, or ~ if they do participate 
~ will be resentful, "looking at their watches" the whole time. 

b. Team evaluation requires a mix of special skills . The process 
requires one or more facilitators able to study and understand 
the process; lead the team through the various steps ~ 
balancing the need to encourage member initiative with the 
need to get the job done; know when and where to seek help; 
communicate clearly; and maintain a positive, productive spirit. 
Other team members must also have similar abilities. These 
include communicating clearly in written and oral form, critical 
thinking, making decisions in a group, finding needed 
information and resources, etc. (While not all members must 
necessarily possess every skill, those skills must somehow be 
available to the team if it is to do the work required.) 
Members lacking such skills can become confused and be 
unwilling or unable to participate. 

c. Team evaluation requires particular attitudes and values . 
This approach to evaluation requires that team members have 
an interest in strengthening the education program, the 
workplace, and workers' roles. Members need to see the 
evaluation process as a potential way of contributing to such 
improvements. Members should value the notion of 
"teamwork" as an asset for both the education program and the 
larger organization. Intrinsic to teamwork are "respect," 
"patience," "perseverance," "listening to others," "compromise," 
and "consensus." Members lacking such positive attitudes will 
likely not feel comfortable on the team and either drop out, or 
interfere with or even sabotage the process. And members 
need to see the education program in a positive light, not as a 
narrow "quick fix" project for "the illiterates." 
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d. Team evaluation requires discipline . Conducting an 
evaluation ~ witii or witiiout odiers ~ is not easy. When a 
team is involved, special attention must be paid to making sure 
that members follow through on the tasks they agree to take 
responsibility for. Otherwise, tasks do not get completed and 
the facilitator or another team member has to take over for 
those not "pulling their weight." To ensure that jobs get done, 
the facilitator has the duty of keeping everyone "on task." 

e. If not properly done, the team evaluation process can be 
skewed to serve the interests of iust one or a few stakeholders . 
If the team is not careful, the process might become skewed to 
focus on issues and recommendations of interest to just one or 
a few stakeholders, rather than the entire group. Some team 
members not accustomed to or interested in the notion of 
"collaboration" might ~ even unknowingly ~ dominate 
meetings with their own personal agendas and effectively 
intimidate or "turn off other members. Some members might 
use meetings as a soap box for personal grievances (perhaps 
not even related to the topic at hand), especially where there 
are no other mechanisms (like a strong union) for airing 
personal concerns. In workplaces where a particular 
stakeholder group normally does the decision-making, it will 
be hard for members from other groups to make their voices 
heard. This is particularly true in workplaces where the 
workforce has a significant number of immigrant workers. 
Their particular language abilities and cultural backgrounds 
might make it difficult for them to "speak up" in a team 
process. 

f. If confidentiality is not respected, the team evaluation can 
needlessly jeopardize the security of those involved . A good 
deal of trust and procedures for ensuring "security" need to be 
in place to make this process work. If those asked to speak 
frankly are not sure that what they say will not be used 
against them or someone else, they will not likely be willing to 
say what is on their minds. 

g. Team evaluation requires special knowledge and procedures 
to minimize bias and ensure validity and reliability . As is true 
with any form of evaluation, the findings of a team evaluation 
could end up being biased, unreliable, or invalid. This could be 
due to team members' bias or lack of experience. 



h. It is difficult to introduce a "team" approach to evaluation in 
a program or workplace not already "team-oriented ." As stated 
under "Strengths" above, a workplace already oriented to team 
decision-making can be a fertile context in which to conduct a 
team-based evaluation. Converely, workplaces (and education 
programs) structured along more traditional lines might not 
understand ~ or might even resist ~ the notion of team 
evaluation. In either case, team members need to be shown 
how to apply team decision-making to the particular task of 
education program evaluation. 



3. Recommendations for those interested in trying team 
evaluation 

a. Evaluation teams need to have or create : 



• Time to do the work 

• Interest : motivation to build a better education program 
and workplace and provide new opportunities for workers; a 
recognition of the value of this evaluation process; a 
willingness to shaie decision-making with others and try 
something new. 

• Expertise to perform the various thinking, conmiunication, 
and other tasks required by the process. 

• A supportive context: a host organization which provides 
an audience for the evaluation and an infrastructure (for 
example, meeting rooms, a computer for data-analysis anc*. 
report-writing, release time for staff to participate) where 
members can reinforce the team skills to be used in the 
evaluation process. 

• A "code of confidentiality " ("tiiist") and procedures for 
maintaining the confidentiality of team activities. 

• An ethic of continuous improvement : seeing evaluation as 
an opportunity for creative, constructive criticism to 
improve the organization and the education program. 

b. To ensure that teams have the above elements in place, 
those organizing teams should use the following team-building 
activities : 
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• Select members carefully. Key decision-makers in the 
organization- which include not only "higher-ups" but 
supervisors who can encourage or inhibit learner 
participation ~ need to be "on board," supporting the team 
even if not actively involved in all its activities. Conversely, 
"trouble-makers" or people with "chips on their shoulder" 
might have to be screened out. 

• Ne gotiate members' roles and those of others (for example, 
co-workers need to know why they are being invited to a 
focus group and groundrules for participating) involved in 
the evaluation. Set groundrules which all members agree to 
follow. 

• Stress the need for continuity within the team and, 
therefore, commitment of members to carry out their 
responsibilities. 

• Provide extra training and technical assistance required to 
develop necessary attitudes and perform particular tasks. 
Team members, for example, might be willing but unable to 
help conduct interviews simply because they never 
interviewed anyone before. Facilitators in various sites 
might develop a "buddy system" among themselves, to 
enable them to give each other feedback and moral support 
as needed. 

• Organize team activities in ways which will allow all 
members to participate actively . For example, translators 
might be necessary for some members, or reading tasks 
might be reduced to allow members with more-limited 
reading abilities to get access to information. 

• Set a "climate " of respect, willingness to listen, and a 
positive vision in all interactions with the team. 

A facilitator might find that, even after carrying out the 
above team-building activities, it appears that it will not be 
possible to form a coherent team. The facilitator should be 
prepared to "just say no" to the notion of conducting a team 
evaluation. She or he might instead suggest that the program 
arrange for an outside evaluator to come in. Or the program 
might postpone the team evaluation idea until a time when 
stakeholders are ready for one. 

c. Teams need to customize, refine, streamline the process to fit 
it to the unique needs and opportunities of each site . Members 
should carefully calculate what they want to accomplish and 
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how much time and expertise they can give to the process. 
Facilitators should present do-able, practical tasks, organized in 
"chewable chunks," which produce clear rewards. Facilitators 
should avoid overly-academic or technical language in team 
meetings, keeping activities short and simple. Tliis is done to 
make the process the most efficient and meaningful one 
possible. At the same time, streamlining must not jump over 
or eliminate elements which are vital to an effective 
evaluation. 

d. Teams should clarify where evaluation fits into the larger 
program planning process . Ideally, teams would begin their 
evaluation work in the early stages of planning the program, 
rather than after the program is already underway. Teams 
should also be clear about the difference between "assessment" 
(normally thought of as measuring individual needs and 
progress) and "evaluation" (a larger process of clarifying what 
is being achieved and deciding further actions related to the 
program). 

e. Teams should consider calling on the expertise of an 
outsider if necessary . While the team evaluation process 
encourages members to take ownership for evaluation 
activities, it is a real possibiUty tiiat some teams will not be 
able to carry out every task required. In such cases, teams 
should consider calling on someone from outside the team ~ 
perhaps a professional evaluator ~ to carry out some of the 
tasks either alone or with other team members. For example, if 
the team wants to generate statistical information and 
manipulate it in a computer database, perhaps someone with 
that expertise might be brought in from the local college to set 
up that system. 

Team members can be the judge of how much 
responsibility they want to give to someone else, so they can 
avoid turning the evaluation into a more-traditional, "outside" 
evaluation in which members would essentially abdicate 
responsibility for the process. If an outsider is brought in (to, 
say, design a questionnaire or conduct focus groups), he cr she 
might show other team members how to perform particular 
tasks so they can do them themselves the next time. 

f. Teams should "get their feet wet" quickly . Teams should 
not be afraid to "jump in" and try collecting some information. 
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analyzing it, reporting it, and acting on it. This can serve as an 
initial trial run which produces both useful data, useful 
expertise, and useful, concrete results. Members should not 
make a "big deal" out of it, dragging out the process, postponing 
unfamiliar tasks out of fear or a desire to "make everything 
perfect." When that happens, teams wiU likely lose or change 
focus and feel they then have to start all over again. Instead, 
members should keep in mind that the more they do it, the 
easier it will be. 

As members gather information, they should review it 
and decide on appropriate actions. Avoid prolonging getting to 
the "action" stage. Otherwise, members might start wondering 
why they are gathering so much information without getting to 
any concrete action. 

g. Teams should understand that their goals for the program 
will change as thev learn more about it . Rather than take this 
as a sign that their original goals were "off target," members 
should welcome this change. It is a sign that the program is 
growing and that the team is trying to improve it by making it 
more relevant. Funders should also recognize this and give 
teams time to continually re-assess their goals as the program 
evolves. 

* h. Teams should develop evaluation procedures consonant with 

the culture of the company and education program . Some 
teams might want a relatively quick snapshot of how the 
program is doing and any problems that need to be attended 
to. In such a case, a team might simply sit down and listen to 
what learners and other stakeholders have to say in one or two 
focus groups. Other teams might want a more-formal, longer- 
term, in-depth study. Such a study could be done through 
ethnographic interviews and observation and come in a 
narrative "case study" format. For those oriented toward "hard 
data," a more-formal evaluation might also present statistics 
acquired through structured interviews and questionnaires and 
perhaps some kind of customized "test" or study of company 
production records or changes in the organization and learners. 

Teams should realize that each of these methods of 
gathering, analyzing, and presenting information has its 
advantages and disadvantages. Narrative information should 
not automatically be dismissed as "lite" or "too soft;" "statistics" 
shouldn't automatically be seen as too technical or insensitive. 
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Teams should also be sensitive to the natural ways 
people communicate and make decisions in the company. In 
many cases, focus groups might be more productive ways of 
eliciting employees' ideas tiian structured, written 
questionnaires. 

Each team should choose the evaluation procedures 
which make sense within the particular culture of the company 
and education program. 

i. Teams should use resources which are familiar and 
accessible . For example, the company or educational 
institution might already use a database program for other 
data-analysis purposes. The education team might get access 
to such a resource rather than have to purchase a new one and 
train people to use it. Similarly, the team might call on the 
company communications department to help prepare an 
attractive version of the team's report. Members might also 
participate in training workshops available through the 
company training department or a local education mstitution 
which could help members develop expertise the team needs. 

j. Teams should balance "product" with "process. " It is 
understandable that most team members will get involved in 
an evaluation with the hope of producing some kind of clear 
"product." In this case, product is thought of as some kind of 
"clear evidence," "data," or "a report" which they can use to 
make decisions with and/or show to the program's supporters. 
Generating useful, readable, relevant information and getting it 
into the hands of intended audiences are important, as they 
provide tangible rewards for the team's work. 

Members should keep in mind, however, that generating 
reliable and meaningful information requires work. They 
should also realize that the very process of sitting members 
down and giving them an opportunity to talk about the 
program has value. Such dialogue can build trust, 
understanding, shared ownership, and other support for the 
program. It can build a team identity and team skills which 
carry over into other aspects of the organization. Teams should 
see these benefits of the team process as "products" of possibly 
greater value than a neatly-packaged set of statistics or report. 
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k. Teams need to be realistic and fair in what they expect from 
an education program . Teams shouldn't expect a short-term 
basic skills program for a few workers to solve problems which 
are longer-term, company-wide, and more-complex in nature. 
Teams should consider doing a comprehensive workplace needs 
assessment, to determine what factors interfere with 
productivity and quality of work life. In tiurn, such an 
assessment can identify the range of initiatives a company 
might take to deal with those obstacles. 

If, for example, employees have trouble reading work 
orders, it might be because the orders are poorly written; in 
such a case, a clear-writing initiative might be in order. If 
workers generally show limited interest in solving problems, it 
might be less due to lack of problem-solving skills than to a 
faulty incentive system; perhaps their interest could be 
increased through an employee-stock-ownership plan. 

Workplace education could thereby be seen ~ and 
evaluated ~ as one of a number of changes an organization 
might undertake to maximize its human potential. 

1. Teams should beware of intra-organizational politics . Teams 
need to remember that they are not operating in a vacuum and 
that the "politics" of the workplace will Ukely impact the ability 
of team members to collaborate. The team process can help 
resolve lingering suspicions and adversarial or "co-dependent" 
relationships by giving all stakeholders opportunities to 
express their views. Members should realize, however, that 
some stakeholders might find it difficult to participate actively 
due not so much to the team process as to the "baggage" they 
carry with them from other relationships they have in the 
workplace. 

m. Funders and other "higher-ups" should show interest in the 
team's activities without interfering in them . Teams often cite 
funders and other "higher-ups" (e.g., high-level officials in the 
company, union, or educational institution) as primary 
audiences for their evaluation reports. If those audiences set a 
positive "climaie ' and demonstrate their interest in what the 
team finds, the teams will likely be motivated to do a good job. 
Conversely, if those "higher-ups" politely ignore the team and 
its findings, the team will have to look elsewhere for its 
motivation. Funders et al should not, however, get overly 
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involved in tiie team and effectively dominate the process or 
distract members from what they want to focus on. 

n. Teams should make recommendations which are realistic . 
Most evaluations will conclude that "if only we had more time 
and money, we could make tremendous strides." While such 
conclusions are probably true, team^s need to be realistic in 
what they recommend in their evaluations. 
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BULL WORLDWIDE INFORMATION SYSTEMS 



Brighton, Massachusetts 



I. PROGRAM BACKGROUND 
Program history 

Bull HN, Inc., is the American wholly-owned subsidiary of the 
French-owned Bull Worldwide Information Systems. The Brighton 
plant is a medium-sized manufacturer of electronic circuit boards 
located in Brighton, Massachusetts. While other manufacturing 
plants here and abroad have consistently shown a loss ~ including 
other plants in eastern Massachusetts ~ the Brighton plant has 
consistently shown a profit and helped to offset overall losses for the 
parent company. 

Competition for cheap and well-made electronic circuit boards 
has become very keen in the international marketplace. The 
Brighton plant is changing its organization of work in order to 
continue to compete successfully. The essence of this change is that 
self-managed work teams assume more and more control and 
responsibility for all phases of work. In addition, the company has 
recently become ISO 9000 certified. This means that workers are 
required to read and follow standard operating procedures contained 
in training and work documents. The combination of the shift to 
self -managed work teams and maintaining ISO 9000 certification has 
made good communication, reading, and writing skills for all workers 
necessary and critical. i 



1 This and other descriptions of the Brighton plant contained in this 
case study refer to conditions from November 1992 to November 
1993, the time during which the team-based evaluation was piloted. 
Since January 1994, conditions at the plant have changed 
significantly. For example, many associates are now required to 
work twelve hour shifts ~ and others work copious amounts of 
overtime ~ in order to complete quick-turn-around orders. With 
associates pressed to manufacture and deUver products as quickly 
and efficiently as possible, education and training appear to become 
less important. Although it is acknowledged that associates need to 
improve their skills more than ever, in the face of rigorous 
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Program goals 



The workforce of the Brighton plant is culturally and 
linguistically diverse. Of the 400 "associates" employed in both the 
"B" Shop (where the circuit boards are made) and the "A" shop 
(where printed circuits are added), approximately 100 do not speak 
English well. The associates are Italian, Portuguese, Cape Verdian, 
East and Southeast Asian, and Asian Indian immigrants. Some of 
these associates have been with the plant for five years or more and 
have a good performance record, but are now being challenged to 
communicate in English at a level that they would not have thought 
possible a few years ago. 

Management is committed to keeping these associates 
employed but the underlying message is indisputable: if you cannot 
keep up with the changes, your job might be in jeopardy and the 
plant might fail. An on-site ESL program seemed a good place to 
begin to provide the kind of educational support for acquiring 
English communication skills that these associates need. It was also 
understood that other associates need other types of basic skills 
instruction and that, in the future, the program might serve them as 
well. The eleven formal goals which the Planning and Evaluation 
Team (PET) generated and which guided the evaluation process are 
described in the Goals Matrix on pages 19-22. 



Description of program 

Twenty-eight associates were enrolled in the ESL program. 
Three classes were offered to accommodate shifts and personal 
schedules. Classes were held twice a week for two hours. If an 
associate attended class during a shift, s/he was given full release 
time. If an associate attended class before or after a shift, s/he was 
compensated at time and a half. 



production schedules, the ESL and basic skills program are poorly 
attended. Associates, supervisors and other managers who were 
very supportive of expanded education programs simply have to 
"put production first." 
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From November 1992 through August 1993, the Bull Brighton 
plant ESL program was partially funded through a grant from the 
Boston Mayor's Office of Jobs and Community Service with 
Training Partnership Act (JTPA) 8% set-aside moneys. The company 
supported the program before, during, and after the grant period. 
The Brighton plant's educational partner is the Adult Literacy 
Resource Institute, a project of Roxbury Community College. 



11. BACKGROUND ON THE TEAM 

History of the team 

The first meeting of the PET took place on November 20, 1992, 
a few months after die program began. David Rosen, the Program 
Coordinator, was familiar with the researchers' team evaluation 
model and had agreed -- with the consent of other team members - 
that the Bull Brighton ESL program would be a good site for a trial of 
the method. He invited researcher Laura Sperazi to the first PET 
meeting, so that she could briefly introduce the evaluation process 
that the team would implement that year. She was to serve as 
facilitator at this site. 

The concepts of working as a team to conduct an evaluation 
and linking evaluation findings to planning were already very 
familiar to Bull employees. Team-based management had 
sufficientiy taken hold within management tiiat most team members 
normally conducted their business in teams and were familiar both 
with setting goals and determining how they would know whether 
their goals were being met. This familiarity witii team procedures 
meant that Ms. Sperazi's introduction of the team-based evaluation 
process was not surprising to most members. As the year 
progressed, team members would disagree to some extent on what 
the evaluation should measure and the time to devote to evaluation, 
but not on the fundamental process. 

Stakeholders represented on the team 
The members of the team included: 

• Jane La Branche, Director of Training 

• Dick Henderson, Director of Personnel 



Ieric 



3 32 



• Manuel Palomino, Training Intern 

• Mike Balas, Plant Manager, "PWA" 

• Ken Manganaro, Plant Manager, "PWB" 

• Alberto Vigianni, Associate and student 

• Reginalda Camaro, Associate and student 

• David Rosen, Director of the Adult literacy Resource Institute, 
and Program Coordinator 

• Jim Ward, Curriculum Developer 

• Melvina Green, Teacher 

• Annie Yu, Teacher 

• Sharon Parmalee, Training Intern, replaced Manuel Palomino in 
September 1993 and was later hired by the company as 
Program Coordinator 

• Ken Robbins, Supervisor, began attending meetings in 
September 1993 



III. THE PROCESS WHICH THE TEAM WENT THROUGH 

Summary: Between November 20, 1992 and October 17, 1993, 
Laura Sperazi met with the Bull PET approximately twice a month for 
two hours. True to the purpose of a PET, team members met to 
discuss a broad array of program planning issues as well as how to 
proceed with the evaluation. As a rule, tiie evaluation would be one 
item on the agenda along witii other program planning-related items. 
However, when the work of tiie evaluation intensified - for example, 
when we were developing survey questionnaires or protocols for 
focus groups - a PET meeting might be devoted exclusively to 
evaluation issues. 

In ttie late spring of 1993, PET members agreed diat a 
subcommittee of tiie PET should focus their attention on evaluation. 
It was getting tedious for the plant managers and otiier company 
personnel to be involved in all the details of tiie evaluation process. 
As a result of the frequency of meetings and the regular Unking of 
evaluation issues to program planning, the evaluation facilitator 
functioned more like a member of tiie team in tiiis program than in 
other settings where the number of meetings was fewer and the 
facilitator role more limited. 
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Between the late spring and early winter the PET prepared, 
gathered, analyzed, informally reported, and integrated into program 
planning information which was gathered in seven different ways: 



• a student survey 

• a managers' survey 

• three focus groups for students 

• two focus groups for managers 

• lEPs (Individual Education Plans) for students 

• a pre-post reading assessment for students 

• informal exit interviews 



The data from these sources indicate that, overall, the BuE ESL 
program achieved most of its goals at least partially. (See Goals 
Matrix, pages 19-22.) The program was poised to expand 
significantly when, unexpectedly, business conditions required a 
major reorganization of shifts and work schedules. As of this writing, 
the ESL program has only a few associates enrolled. 



Phase I : Initial preparations 

Program coordinator David Rosen introduced team-based 
evaluation to the PET before he invited researcher Laura Sperazi 
(who was to serve as facilitator at the site) to talk with the group. He 
was better prepared to make such an introduction than other 
coordinators might be because he had worked with the researcher on 
a prior team-based evaluation project. 

In addition, Ms. Sperazi (hereafter referred to as "the 
facilitator") prepared for working with the team by taking several 
tours of the plant and discussing issues related to the ESL program 
with the Training Director, the Engineering and Training Intern, the 
teachers. Curriculum Developer and Program Coordinator. The 
facilitator also wrote a letter to the Training Director and the 
Training Intern which explained the history and purpose of team- 
based evaluation. She then prepared materials for team members 
which outlined the steps of the evaluation process and a possible 
timeline. 
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Phase II : Planning the evaluation 



It took roughly six months for the team to plan its evaluation. 
During that time it (1) revised its program goals several times; (2) 
clarified the differences between evaluation and assessment; (3) 
grappled with the fact that it had not collected base-line data of any 
kind other than a series of "placement" assessments in reading, 
writing and listening; and (4) tried to reach consensus on what the 
focus of the evaluation should be. 

Before it settled on a focus and data collection methods, the 
team also fortified itself in two additional ways: 

First, selected team members met with members of two other 
PETs (from the Haartz Corporation and the Norton Company) to share 
ideas about how to focus an evaluation of a workplace education 
program. 

Second, anticipating the need to present the results of their 
evaluation in statistical form, selected members participated in a 
day-long training on the use of the Statistical Package for the Social 
Sciences (SPSS), a database program. (Steven Andrews and Alice 
Oberfield Andrews conducted the training and were available for 
telephone consultation afterwards.) Although some of the SPSS 
training focused on actual manipulation of the database, much of it 
focused on how statistical analysis is only as good as the information 
collected. This further stimulated the team to think through the 
focus of their evaluation ~ a question which team members had 
much on their minds as they entered the training. In the end, the 
team chose not to use SPSS for data analysis. The data which the 
team collected did not warrant such analysis. There was some 
discussion about additional training for Bull support staff in the use 
of the database so that it could be used in the future, when the data 
warranted it. 

Clarifying program goals and the information to look 
for 

The program goals which the team fmaUy agreed on for the 
purposes of the evaluation are presented in the Goals Matrix on 
pages 19-22, along with an assessment of whether they were 
achieved or not. 
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Questions about the kind of information the team needed to 
collect about the ESL program in the short ar^id long term are best 
summarized by the events of the SPSS training. 

Jane La Branche, the Training Director, came to the SPSS 
training with questions about the evaluation's direction to date. 
Basically, she said that at this point in tiie development of the 
program (five months), it was unfair to measure the success of the 
program by how well it meets long-term organizational needs. She 
said that, at this time, the heart of the evaluation needs to be on 
measuring and documenting the improvement of participants' 
language skills. What level of language skills are needed to help 
bridge associates to other training? How fast can the company 
expect associates to approach this level of skill? Are associates 
progressing? 

At some point, she said, it would be appropriate to determine if 
the program is having an effect on the organization. She said that 
the team should think about these issues now, in order to be well- 
prepared to evaluate them down the road. For now, however, the 
focus of the evaluation should be on improvement of language skills. 
What the team learns about improving language skills should^ be 
used, in turn, to improve the program. 

Jane's comments got everyone thinking again about the kind of 
evidence it is important to gather both to help Jane argue effectively 
with management for continued support of the ESL program and to 
improve the program. In fact, the team shared a tacit assumption 
that the evaluation should expose opportvmities to improve the 
program ~ as opposed to determine whether the program should 
continue or not. (Some team members later said that they felt 
artificially driven by grant requirements to evaluate program and 
student outcomes when, at this early stage of program development, 
outcomes were not their main concern.) 

The conceptual model that the team agreed they were working 
from looks like this: 
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Language Skills/Competencies 



Participation in 
the ESL Class 



Organizational 
Outcomes 



\ "Willingness" ^ 
(to learn and to change) 



This model assumes a connection between acquisition of 
language skills and competencies and organizational outcomes. It 
also assumes a connection between "willingness" to learn and to 
change and organizational outcomes. "Willingness" was intended to 
describe those personal, affective changes which occur when people 
enjoy learning and which affect organizational outcomes in the long 
term. The group decided not to try to measure "willingness" at this 
point. Rather, it would focus on improvement of language skills and 
competencies. 

The team made the following suggestions about the focus of the 
evaluation at this time: 



(1) Revisit the Placement Assessment (which Annie Yu, 
one of the teachers, designed and which all program 
participants have taken) as a pre- test which can be given 
again as a post test, and which can be scored in a way to 
demonstrate individual progress over time. The point here 
is that there is a system already in place that can be built on to 
provide more complete pre- and post information on all students 
than is currently available. 

The SPSS trainers had some specific suggestions for designing 
the post test based on tjie placement instrument. ^ 



2 These suggestions included the following and were based on 
designing post-tests for reading, listening, speaking and writing: (a) 
Score each part of the test so that the final score is more than one 
number; (b) It may be useful to score the final test as one number as 
well, and that number can be a range from, for example, 1 to 3. This 
means that if an associate scores a 4, s/he is ready to "bridge" out of 
ESL to another training. The point here is that it is necessary to 
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(2) Design a pre/post test using the most important 
competencies that Jim Ward, the Curriculum Developer, is 
already in the process of identifying. Again, the point is 
that there is a system about to be put into place that can be built 
on to provide more complete pre- and post- information on all 
students than is currently available. There seemed to be 
agreement in the training that this would be a logical ~ and fairly 
easy ~ thing to do. 

(3) Assess model associates to determine their level of 
skill as a standard for participants in the ESL program to 
reach toward. Jane had expressed interest in knowing what 
level of language is sufficient for workers to participate in other 
training programs. The SPSS consultants described this task as 
"definitional." The strategy suggested to define ~ even in broad 
terms ~ the level of skills necessary to perform well was: assess 
model workers with the same or similar assessments used with 
ESL program participants. This will give the staff and Jane a good 
idea of what the program should be working toward. 

(4) hiclude a student self-report as part of assessment of 
progress. Team members agreed that students' reports of their 
own learning are an important part of a complete evaluation. Mel 
Green, a teacher, has begun collecting students' self reports in 
class, with a particular focus on how to improve the program. The 
team members present at the training suggested that the student 
self report might be part of the lEP. 

(5) Gather systematic supervisor feedback on the effects 
of the program on work. Although much of the training 
focused on measuring progress from the teacher's perspective, 
some time was spent thinking through the value of getting 
supervisors' perceptions of how class learning is translating into 
changes in behaviors at work. A simple survey could be the first 
link in the evaluation system which ties to organizational effects. 



identify when associates have developed their skills enough that 
they are ready to leave ESL and move on to something else ; (c) 
Huninate the choice of stories to read in the post test. Standardize 
the stories used in the post test to match the levels of the stories 
used in the pre-test. 
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Even a small survey could provide data over time which would 
supplement the more skills-based data, and provide Jane with 
stronger arguments for '^ontinued company support of the 
program. Jane said that this type of survey would not provide 
sufficient evidence of program success on its own, but would be 
meaningful when combined with other data. Team members 
present at the training also discussed the possibility of surveying 
other people like associates not enrolled in the program or other 
managers. These people would simply provide additional 
information on the extent of associates' improvement in English; 
they would not constitute a control group. 

At this point in the evaluation it was important for the team to 
make choices about next steps in Ught of the resources available. 
How much time and creative energy are needed to move forward on 
each of the recommendations/actions outlined above? What are the 
team's priorities? 

The team reviewed another set of materials before it made 
final decisions about where and how to focus its evaluation. These 
materials are the seventeen quality indicators for workplace basic 
skills programs which were developed by the Massachusetts 
Workplace Education Initiative. The indicators (Appendix A) identify 
and set standards for the essential components of a quality basic 
skills program. 

When the team reviewed these indicators in March, members 
felt that it would be overwhelming to entertain all seventeen 
indicators. They agreed that the team should set aside a whole day 
to review the indicators and assess the program against them. (The 
team did this in late August.) Members agreed that, for the present, 
the evaluation should consider one indicator ~ the extent to which 
the program is providing a quality learning environment for the 
associates. 



Designing a strategy for gathering data 

The overall strategy which the team developed for gathering 
data was: 
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(1) The team discussed the issues until it clarified the kinds of 
information it wanted to collect (as described above). 

(2) With additional input from the team as needed, the evaluation 
facilitator developed a draft instrument to collect the desired 
information. 

(3) The team would then review and revise the instrument. 

Often the process of revision took place in three or four phases. 
The team would review what the evaluator had done; the evaluator 
would then revise the draft and submit it for review again. 

In the late spring team members agreed on a strategy for 
gathering the data it wanted. There were still differences among 
team members about whether it was desirable to assess 
decontextualized communication skiUs (reading, Hstening, speaking, 
and writing as stand-alone phenomena) but it seemed fair to try to 
develop a way of assessing improvement in writing skills (using the 
placement assessment as a base-line) as one part of the evaluation. 
(Possible use of the BEST was discarded after consulting with experts 
about the limited value of available standardized tests.) 

Student and manager surveys and focus groups would allow 
the teani to gather information both on the perceived quality and 
appropriateness of the learning environment being provided and the 
changes in students' work and personal lives which result from 
participation in a company-sponsored ESL course. The team also 
agreed to continue specifying competencies for the lEP for teachers 
in subsequent classes. These competencies would include both work- 
specific and general communication competencies. The team also 
suggested at this time that formal exit interviews of key staff and 
students be conducted by the evaluation facilitator. (The facilitator 
was able to conduct only informal interviews at the end of the 
project. See Goals Matrix, on pages 19-22.) 

There was one more resource available to the Bull team for 
input on how to think about evidence, measurement, and instrument 
design in general, and how to develop a post-test based on the 
placement assessment. That resource was Donald Cichon of Donald 
Cichon Consultants. The team had been coming to the conclusion that 
It would not be possible to create a meaningful post-test given the 
lack of quantifiable base-line data in reading, speaking or listening 
but that it might be possible to do something meaningful with 
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writing. He met with the team and made suggestions about how to 
design and score a writing post-test. 



Phase III : Gathering, organizing, analyzing, and 
reporting the information 

Gathering and organizing the data 

The Curriculum Developer, and later the Training Intern, along 
with the evaluation facilitator were responsible for carrying out the 
data-gathering activities. For example, the Curriculum Developer 
oversaw the administration of the student survey as a class activity 
in all three classes and distributed the managers' survey to the 
appropriate managers. The Training Intern conducted background 
interviews for the student survey and the first supervisor focus 
group. The evaluation facilitator conducted three student focus 
groups and one supervisor focus group. 

Once the data were gathered, the Curriculum Developer, 
Training Intern, and evaluation facilitator were also responsible for 
organizing the data. For example, the Curriculum Developer provided 
a simple tally of the frequencies of responses to the student survey. 
The Training Intern provided a simple tally of the frequencies of 
responses to the student background survey and organized a 
summary of the discussion points of the first supervisor focus group. 
The evaluation facilitator organized a summary of the discussion 
points of the three student focus groups and the second supervisor 
focus group. 

Analyzing the data 

Once the data were organized, the facilitator offered an analysis 
of the data to the team and the team would discuss that analysis ~ 
always enriching the analysis in ways that only staff "inside" an 
organization can. 

For each of the six data-gathering activities which the team 
undertook, and which are outiined below, are additional infonnation 
on the type of data gathered, its organization, and the team's 
analyses. 
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Student Questionnaire 



The purpose of the student questionnaire was two-fold: to 
assess the extent to which the students believed the program was 
providing a quality learning environment, and to assess self- 
perceptions of change m communication skills at work and in life 
outside of work. 

In early March, the Training Intern conducted informal 
interviews with ten students in order to gather information with 
which to develop survey item.s. This information focused on 
identifying the standards of a quality learning environment from 
the students' perspective. (See Appendix B.) After the team 
developed the student questionnaire, the Curriculum Developer 
piloted a draft of the questionnaire with two students who were 
also members of the PET to ensure reliability and validity of the 
instrument. (See Appendix C.) 

Twenty of the twenty-eight students enrolled at the time 
completed the survey. The Curriculum Developer organized the 
raw scores so that the facilitator could easily interpret them. (See 
Appendix D.) 

Students assessed the quality of their program overall as very 
high but did not assess their changes in communication skills very 
high. Two issues emerged that generated particular interest and 
curiosity within the team: ( 1) students appeared to say that the 
curriculum focused too much on work-related issue, and (2) in a 
program that they praised so highly, students self-reported 
relatively little change in their communication skills. These issues 
were followed up on and clarified in focus groups. 



student Focus Groups 



The student survey was followed by three student focus groups 
- one for each ESL class. The facilitator conducted these groups 
with support from a new ESL teacher during three class periods.^ 
(See Appendix E.) The focus groups clarified the two issues 
identified above and provided important information about the 
students' feelings about computerized instruction. 

First, the students did not think that the curriculum focused too 
much on work-related issues. They found the question on this 
topic in the questionnaire confusing and did not answer it to 
reflect their intent. The focus group clarified that students 
understand very well the importance of a work-related 
curriculum. They understand tliat the future of the company 
depends on how flexibly they can perform their jobs, which 
depends on how well they can communicate in English. 

Second, the students were very harsh in their assessments of 
their improvements in communication skills because tliey do not 
feel that they can speak at the level at which they want to speak. 
They are impatient to achieve their life-long communication goals. 
They want more instruction in speaking skills. They want to be 
able to practice their English with a proficient English speaker, not 
another non-native speaker of English who cannot speak English 
well. 

Third, students looked favorably on computer-assisted 
instruction but did not want it to replace real conversation with a 
teacher. 



3 The evaluation facilitator conducted the student focus groups on 
the first day of class with the new ESL teacher. The focus groups 
proved to be a wonderful introduction to both the students and 
program for the new teacher. However, had the teacher been 
teaching the class ~ even for a short period of time ~ her presence 
would have compromised the focus group process. The sUidents 
needed to feel that they could speak freely and honestiy about their 
program. This would have been difficult or impossible with a well- 
known teacher in the group. 
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Managers' Survey 

A total of four supervisors completed the managers' survey. It 
was distributed by the Curriculum Developer and results were 
tallied by the facilitator. The survey contained twenty items for 
which the supervisor was asked to assess change in each associate 
under his/her supervision from the year prior to ESL class (1992) 
to the current year ( 1993). The four supervisors completed a 
total of fifteen surveys. There was a total of twenty-eight 
associates enrolled in the ESL program so thirteen associates are 
not represented. (See Appendix F.) 

Of primary interest was whether the supervisors witnessed 
any change and, if so, on which items. The results are striking. 
Most supervisors report very little change: 

• For three associates, supervisors report no change at all. 

• For two associates, supervisors report change on only one 
item. 

• For another two associates, supervisors report change on 
three items. 

• For four associates, supervisors report change on six items. 

• For one associate, a supervisor reports change on nine items. 

• For one associate, a supervisor reports change on thirteen 
items. 

• For one associate, a supervisor reports change on fourteen 
items. 

• For one associate, a supervisor reports change on seventeen 
items. 

The obvious question was why the supervisors noted so little 
change. Was this a reflection of the poor quality of the ESL 
program? Or were there other factors at work that deserve 
attention? For example, do the supervisors have enough contact 
with the associates to be able to make an observation about 
change in communication skills? And do the associates have the 
opportunity to apply all the skills they acquire in the classroom to 
their work on the floor? The managers' focus groups attempted to 
answer these questions. 




Managers' Focus Groups 



There were two focus groups with managers (also referre to 
as supervisors). In addition to providing an opportunity to ; .th 
information from the supervisors, the focus groups were al' se' a 
as an opportimity to educate management about the ESL ar >. basic 
skills program and to enlist their greater support of and 
involveme- t in it. (See Appendix G.) 

Three supervisors attended the first group (which was 
facilitated by the Training Director) and four attended the second 
(which was facilitated by the evaluation facilitator). In general, 
supervisors corroborated the results of the survey. They did not 
see much change in the associates who were enrolled in the ESL 
program. The reasons for this are varied. 

First, supervisors do not have a lot of one-on-one contact with 
associates in the course of a week and do not have the 
opportunity to assess the kind or quality of change in 
communication skills that is taking place. Second, as a rule, people 
whose first language is not English tend not speak in English at 
work. Even if their English language skills are improving, the 
culture of the floor does not encourage associates to speak in 
English. The opportunity to witness any change is thereby 
diminished further. Supervisors stressed that all the associates 
are "good performers" and that "good performance is not the 
issue." 

At the same time, it was easy to see some dramatic changes in 
the few associates who do "interface" with supervisors and 
engineers and who are exceptionally motivated. 

Supervisors agreed strongly that there was a need for more 
communication in English on the floor. The management focus is 
on developing a self-managed workforce. There was consensus 
that the company is still "light years away" from having a self- 
managed work force and that making English the common 
language of work on the floor is essential. Supei*visors agreed that 
the ESL class should stress self-initiated conversation in English. 
Supervisors expect that the ESL staff will stress the importance of 
speaking in English for interpersonal communication and for work. 
Supervisors underscored that it is a false expectation that people 
will speak English on their own when it is not their first language. 



45 

16 



Structures must be in place to encourage and support them both 
in the ESL program and on the floor. 

With hindsight, the results of these focus groups highlight how 
critical it is to clarify supervisors' expectations for what a 
workplace education program will achieve when the program 
starts. Program staff can educate supervisors about realistic 
expectations; but, often, program staff also need to learn what 
these expectations are themselves - as was the case at Bull. 

Supervisors also made the following points: 

• Supervisors should become part of the lEP process. 
Supervisors agreed that being part of the lEP process was a 
good idea. It gives a supervisor a good basis for evaluating 
what a person can accomplish. There was a lot of discussion 
about how associates' personal goals should also be integrated 
into the lEP and shared with the supervisor - so the 
supervisor can support the associate in achieving those 
personal goals. There was also agreement that the associate 
should feel at the center of the process - making choices about 
what to share, what to commit to, etc. 

• The curric ulum should take a broad view of what is "work- 
related." The ESL program can support supervisors and the 
company's overall well-being by bringing issues of "quality, 
cost, and delivery" into the classroom and curriculum. The 
program should emphasize that "people are responsible for 
their past, present, and future," not to "get people nervous," but 
to show that "the company needs 100% support." The program 
should give specific examples of why it is important to develop 
a certain skill. The focus of the curriculum should be: what 
facilitates team-based work? There should be discussions 
about what a self-managed team is, and the problems that 
associates anticipate they will encounter in this new mode of 
work. 

• Diversity issues shou ld be addressed in the curriculum. The 
question is: How does the message that you can't do something 
get conveyed to associates, and what role does cultural 
diversity (racism, prejudice, ignorance) play in the sending and 
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receiving of that message? In addition, diversity issues keep 
the students engaged. Since it is true that the ESL curriculum 
"can't do everything," diversit>^ training is something that the 
company might offer as a refresher course in diversity issues 
for everyone, and then the ESL curriculum can support it. 

• Regular focus groups : Supervisors agreed that it would be 
good to have a half-hour focus group between supervisors and 
ESL staff every six to eight weeks to keep up with the program. 

Writing Assessment 

The results of the writing post test demonstrate that associates 
made progress in writing during their enrollment in the ESL 
program. 

Individual Education Plans 

In order to ensure that the competencies in the lEPs are the 
competencies which management believes to be the most 
important, the Curriculum Developer surveyed supervisors and 
asked them to rate the priority (high priority, priority, low 
priority) of fifty items which might be incorporated into the lEPs. 
(See Appendix H.) 

The lEPs for the grant period demonstrate that associates 
achieved the desired competencies in the ESL program. See Goals 
Matrix, below. 



The Goals Evaluation Matrix 

The following Goals Evaluation Matrix presents the program 
goals along with the data-collection procedures used to collect 
information related to those goals. On the basis of the information 
collected from the sources just described, an assessment is made 
about whether the goals were achieved or not and, if they were 
achieved, to what extent were they achieved. An asterisk indicates 
that the assessment needs some explanation. The explanation is 
provided at the end of the Matrix. 
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Bull Workplace Education Program 
Goals Evaluation Matrix 

• = This goal was evaluated 

A = This goal was achieved, as Indicated by the related measure 

AP = This goal was achieved partially, as indicated by the related measure 

NA = This goal was not achieved 

* = See notes at end of Matrix INSTRUMENTS/PROCEDURES FOR 



DOCUMENTING AND MEASURING GOALS 



GOALS 






of ^ 

^ 




'A 
{\ 

■C 


^ i 

§W 


1 . Provide appropriate learning arrange- 
ments to associates in the program. 


•A 


• A 










2. Identify and enable associates to 
acquire oral communication (listening 
and speaking), reading, writing, and 
math skills needed to participate in 
self-managed work teams, collaborate 
in data collection, and conduct themselves 
productively in meetings. * 1 


• AP 


• AP 


• AP 


• AP 


• AP 


• AP 


3. Enable associates to be ready to 
participate in Bull's standard computer 
training course through: 
a. helping associates acquire 
necessary listening and speaking skills, 
and basic knowledge of what computers 
can do, and 


• A 








• A 




b. providing suggestions to computer 
training course instructors on how they 
can help limited English speaking 
students who enroll in their course to 
succeed. 














4 Identify and help associates acquire 
listening, speaking and writing skills 
needed to participate in other Bull 
training courses and in on-the-job training 








• AP 


. AP 





IS 



INSTRUMENTS/PROCEDURES FOR 
DOCUMENTING AND MEASURING GOALS 



GOALS 


^^/^ 




' ■ 

■ ^ 

■\><^'^^ 
K ,o 

^l-l 




> 




o. tnaDiG associates lo reaa, wriie 
and interpret documentation needed 
Tur Uaiiy worr\ v^'y* o.w.L/. o, uiuy 
prints, etc.) 

a. as indicated in ISO 9000 guidelines 


. AP 


. AP 


. AP 




. AP 




b. for safety (e.g. Right to Know Law, 
reading M.S.D.S. and safety signs) 

c. for quality 















d. to understand company policies 
and company communications 














o. rieip associates acquire iisiening, 

crioaUnn roaHinn anH \A/ritinn qWIIIq 

needed to participate in the Performance 
Appraisal Management Process 

^ infprv/iPXA/inn 

Cl* II lid VIC W II 1m 














b. reading and understanding 
performance appraisal forms 














c. assessing their goals 










• AP 





d. negotiation skills 














e. peer assessment skills (listening and 
speaking skills needed for commu- 
nication with one's "internal customers") 














7. Classroom activities should model 
iric Ucnavioro inai iiidiiciyciMcru diiu 
employees at Bull are trying to achieve 
in the organization (i.e. teamwork, open 
communications, etc.) 














8. To enable associates to achieve their 
personal learning goals, related or not to 
the workplace. *2 


• AF 


) 


• AF 


> 


• AF 





INSTRUMENTS/PROCEDURES FOR 
DOCUMENTING AND MEASURING GOALS -.a 



GOALS 



^^^^ 



9. To enable participants to achieve their 

''^ciiimiy vjudlo do olctlt/U III int/U inuiVIQUal 

Education Plans through the application 
of a competency-based curriculum. We 

eXDect 80% or mnrp nf thp nartioinantc tn 

attain 80% of the competencies outlined 
in their Individual Education Plans within 
the grant period. 










• A*3 




10. Furthermore the program will: 

a. establish itself as a program that will 

continue and develop beyond the 

period of grant funding 












. A 


b. establish, maintain and improve a basic 
skills needs assessment 












• AP 


c. establish ah ongoing planning arid 
evaluation team 












• A 


d. take fuTl advahTage oif BulTs compute^^ 
lab by offering computer-assisted 
basic skills instruction 












• A 


e. foster in ali employees a mindset of 
lifelong learning, continuous learning 


• AP 




• AP 




• AP 




1 1 . The program will build a strong 
education and training partnership 
between Bull, Roxbury Community College, 
and the community-based education 
programs supported by Roxbury 
Community College's Adult Literacy 
Resource Institute 












• NA 
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INSTRUMENTS/PROCEDURES FOR 
DOCUMENTING AND MEASURING GOALS 



GOALS 



^0 
•r- "J 

5g 



0\ 



\ "5S' 

KIT 



1 1 . continued 

a. To develop a referral relationship 
with the Jackson Mann Community 
Center's adult learning program and with 
Roxbury Community College's Division of 
Continuing Education and Community 
Services ESL programs, to serve associates 
who have earning needs beyond the current 
capacity of the program, or associates who 
want to meet their learning needs outside 
of the work environment. 



b. To develop a long-term referral relation- 
ship, and possibly continued worksite 
programs with the previously named 
centers for associates who may need 
continuing education (GED or External 
Diploma Programs) *4 



NA 



AP 
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Notes on the Matrix 



The following notes summarize the comments of the members of the 
planning and evaluation team who worked most intensively on the evaluation, 
and who reviewed the goals matrix after the evaluation was concluded. 

*1 . This goal is too big and therefore difficult to assess. The skills should be 
broken out so that listening, speaking, reading, writing and math are 
evaluated separately. 

•2. If this goal had been phrased "work toward their goals" instead of "to 
achieve their goals," it would have been fully achieved. It was impossible for 
most associates to achieve their goals in a ten-month period because their 
goals were very ambitious and would take years to achieve. However, all of 
them did work toward their goals with success. 

*3. The team does not have full confidence in this measure. The teachers -- 
who rated the lEPs -- were fastidious in their record keeping and can 
subjectively substantiate their assessments of students' progress. But they 
were, in fact, developing the lEP process as the grant year progressed. This 
means that they did not have a fully developed process in place at the 
beginning of the program which they followed, with base-line data against 
which they made an objective assessment of achievement of competencies. 
At the end of the grant year they did have a process in place that they felt 
very confident about and which subsequently-hired teachers are now using. 
After the evaluation was completed, two members of the Planning and 
Evaluation Team commented that this might be a reasonable way for a new 
program to proceed. 

*4. Educational services were expanded internally - not in relationship with 
community agencies - to include ABE and GED. 
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Preparing a strategy for reporting the findings 

At the time that the evaluation was conducted, the evaluation 
team was the primary audience for the results of the evaluation. 
Members understood that the evaluation system was just getting 
worked out. They felt that they could use their preliminary findings 
internally to develop the program. However, there was an 
understanding that, eventually, evaluation results could be used to 
argue for program continuation and expansion. The team wanted to 
first critically review the information being gathered, and their 
interpretations of it, before they reported results with any 
confidence to management or other audiences. 

Within the team itself, members communicated what they 
were learning from data-gathering activities in both written and oral 
formats. The facilitator took primary responsibility for summarizing 
the results of particular data-gathering activities for other members 
to consider. 

For the team's external audiences, the Curriculum Developer 
authored a program "Annual Report" which drew on the team's 
findings. 



Phase IV : Deciding what happens next 
Taking follow-up action 

Based not only on the results of the evaluation but on the 
integrated planning and evaluation process in which the team had 
engaged all year, with the support of management, the team decided 
to: 

( 1) Expand the program to include native English speakers at the 
pre-GED and GED levels. 

(2) Integrate a computer-assisted instructional program 
purchased from Josten's Learning Systems into the newly 
expanded program. 

(3) Involve supervisors more intimately in the development of 
lEPs so that the lEPs better reflect the students' work-related 
needs, and the supervisors better understand the students' non- 
work -related goals. 
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(4) "Sell" the education program to supervisors and students as 
part of the company's strategy for organizational change -- a 
change which everyone understands is partly underway and 
whose pace will accelerate. 

As mentioned earlier, although the Jostens system has been 
installed, enrollment in the program is very low. Intensely 
demanding production schedules have superceded the 
implementation of an expanded education program. 



Evaluating the evaluation 

Team members and the facilitator commented that there are 
two directions which the evaluation might have taken that would 
have strengthened it. These are: 

• The team could have started its process of identifying goals by 
referencing the goals of the program as written in the grant 
proposal. While some of those goals were included in the revised 
goals list, it still took six months to complete the new list and 
begin to collect data. If the team had begim their work with the 
goals they had in hand, they could have still continued to refine 
goals and also begun to collect information. 

• At the same time, some team members regretted having been 
too driven by the outcomes of the grant proposal - outcomes that 
are more-realistically addressed in the third - not the first -- 
year of a program. 

• Earlier data collection would have been useful. The evaluation 
facilitator could have directed the goal-setting process more 
firmly and guided earlier data collection. Earlier data collection 
would have given team members an immediate feel for how 
formative evaluation can assist program development - 
something that they could have built on during the year. 

• It would have been useful to have deeper discussions early on 
about the differences between assessment and evaluation, and 
about what team members wanted to learn from an evaluation. 
In retrospect, team members had different expectations about 
what they wanted an evaluation to demonstrate about their 
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program. For example, the Training Director was clear about 
wanting evidence of improvement in reading, writing, and 
speaking against a base-line measure. The two teachers in • .e 
program were unconcerned with base-line measures and fc th. i 
student self reports and teachers' observations were adeq^ e 
methods for documenting progress. 

The goal-setting process only partially elucidated these 
differences. But these differences would nonetheless sur;Ace in 
the ways that teachers, other education staff and managers 
discussed <waluation goals and projected activities. The 
evaluation would have been stronger if those differences had 
been explored more fully early in the evaluation process, and 
proactively rather than reactively. 

• Team members agreed that the presence of an evaluation 
facilitator probably kept a firmer focus on evaluation activities 
than had they worked entirely on their own. However, team 
members (especially the Director of Training) were confident that 
the team would have evaluated on their own. After this pilot 
year, they are confident that they have some evaluation processes 
in place on which they can build. 



IV. REFLECTIONS 

Facilitator Laura Sperazi"s comments: 

• Overall, team-based evaluation worked well in this company. 
Bull employees are accustomed to working in teams so the 
evaluation team format was familiar. At the same time, their 
familiarity with teamwork required the facilitator to sit back and 
let the team govern the process in a way that was more 
synonymous with their culture and language than was her 
prepared outline of evaluation activities. 

• The facilitator conducted more of the actual evaluation work 
than anticipated. Doing the work justified her presence on the 
team and also allowed her to model sortie "evaluation behaviors" 
that some team members learned from. 

• This team would have conducted some type of evaluation 
without the facilitator. The Program Coordinator is very 
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knowledgeable about evaluation and would have required that 
some evaluation activities be carried out. The company culture 
also values (even demands) evidence of effectiveness of training 
programs. The Training Director would have required that some 
evaluation activities be carried out. Without the facilitator, this 
team would have been a good audience for a Guide on team-based 
evaluation of workplace basic skills programs. 

• Waiting to collect data until the goals were completely defined 
made the activity seem more momentous than it should be. Early 
data collection helps the team to be unattached to the data. Some 
of it might be good; some not so good. Waiting until all conditions 
are (relatively) "perfect" or complete puts more pressure on the 
data-collection process than is necessary. 

• It is important to note that several times during the course of 
the evaluation, and again at the last meeting where some 
members of the evaluation team gave the facilitator feedback, the 
following perspective on developing and evaluating a new 
workplace basic skills program was articulated: 

It takes the experience of developing and attempting to 
evaluate a program over the course of a year to know what 
it is that the program might accomplish and how it might 
best be evaluated. While there is a lot of pressure on the 
program to know what it can accomplish at the beginning, in 
reality, in this new field of workplace education which has 
no set curricula and established evaluation tools, it takes a 
full cycle of course development and evaluation to work out 
clear goals and evaluable indicators and to design 
appropriate evaluation proceaures. 

Therefore, the "failure" to produce a complete evaluation 
system at the start of a new program is really not a failure. It is a 
function of the state of the art of program development. Spending 
the better part of the year openly learning what the program can 
do and how it might be evaluated best may well ensure the 
subsequent success of the program. Similarly, holding up the start 
of a program until all base-line measures ~ and all other 
programmatic bells and whisties ~ are in place may be not only 
unrealistic but also not in the best interest of the students or the 
company. 
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Sharon Parmalee 3/28/93 

ESL Student Survey for developing a tool to assess ESL 
classes 

1. Characterize an ideal classroom at Bull 

2. Characterize a bad classroom at Bull 

3. Characterize a good ESL teacher 

a. What would s/he be doing? 

b. What would s/he be thinking? 

c. What would s/he be feeling? 

4. 3. Characterize a bad ESL teacher 

a. What would s/he be doing? 

b. What would s/he be thinking? 

c. What would s/he be feeling? 

5. Characterize an ideal ESL class at Bull 

a. What would sbe happening? 

b. What would people be doing? 

c. What would people be thinking? 

d. What would people be feeling? 

6. Characterize a bad ESL class at Bull 

a. What would sbe happening? 

b. What would people be doing? 

c. What would people be thinking? 

d. What would people be feeling? 
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Bull ESL Program Code # 

Student Survey June 1993 

Instructions: 

We want to know what you think about your ESL class, /e 
will use your answers to improve the program next yes . 
Please draw a circle around the answer that is closest to the 
way you feel. 

Your answers are confidential. 

1 . The class helps me communicate on my job better. 

always usually sometimes never 

2. I have enough time in class to learn and make progress. 

always usually sometimes never 

3. Class is hard for me. 

always usually sometimes never 

4. Class is easy for me. 

always usually sometimes never 

5. There are too many lessons about our work. 

always usually sometimes never 

6. The books, worksheets, and other handouts the teacher gives us are 
interesting. 

always usually sometimes never 

7. The books, worksheets, and other handouts the teacher gives us are hard for me. 

always usually sometimes never 
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8. The books, worksheets, and other handouts the teacher gives us are easier 
than I like. 



always 



usually 



sometimes 



never 



9. These are questions about your teacher. Answer them the same way you 
answered the previous questions. The only difference is that these questions 
do not have numbers next to them. 

My teacher: 

• talks more than I like. 



always 



usually 



sometimes 



never 



• gives me a chance to talk in class. 

always usually sometimes 



never 



• encourages me to ask questions. 

always usually sometimes 



never 



• cares that I am learning, 
always usually 



sometimes 



never 



• helps me when I don't understand. 

always usually sometimes 



never 



• gives more homework than I like, 
always usually 



sometimes 



never 



• gives less homework.than I like, 
always usually 



sometimes 



never 
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• speaks too fast for me. 
always usually 

• speaks clearly. 

always usually 

• uses words I understand, 
always usually 

• treats me as an adult, 
always usually 



sometimes 



sometimes 



sometimes 



sometimes 



never 



never 



never 



never 



10. I feel comfortable telling the teacher I don't understand, 
always usually sometimes 



never 



1 1 . Please draw a circle around NOT MORE THAN FIVE things that you would 
like to have more of in your class. 

• I'd like more: 



- exercise books 

- dictionaries 

- job-related materials 

- stories 

- newspaper articles 
" math 

- listening exercises 

- speaking 

- reading 

- writing 

- is there anything else you would like more of ?. 
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12. These are questions about your class. 
My class: 



• makes me want to continue learning. 

always usually som ttlmes never 

• helps me when I'm not at work. 

always usually sometimes never 

• helps me feel more confident. 

always usually sometimes never 

• helps me to achieve my personal goals. 

always usually sometimes never 

13. These are questions about using your English skills at work. 

• LAST YEAR, before I started class, I READ AND UNDERSTOOD 

work documents (for example, S.O.D. s, M.S.D.S.s, blueprints, company policy.) 

always usually sometimes never 

• LAST YEAR, before I started class, I SPOKE English at work (for 
example, to co-workers, to supervisors.) 

always usually sometimes never 

• LAST YEAR, before I started class, I WROTE at work (for example, 
memos and notes.) 

always usually sometimes never 
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• LAST YEAR, before i started class, I UNDERSTOOD what people at 
work said to me in English (for example, at department meetings, in my work 
area, at business meetings.) 



always usually sometimes never 

•NOWIREADAND UNDERSTAND work documents (for exar jle, 
S.O.D. s, M.S.D.S.s, blueprints, company policy.) 

always usually sometimes never 

•NOWISPEAK English at work (for example, to co workers, to 
supervisors.) 

always usually sometimes never 

• NOW I WRITE in English at work (for example, memos and notes.) 
always usually sometimes never 

•NOW I UNDERSTAND what people at work say to me in English (for 
example, at department meetings, in my work area, at business meetings.) 

always usually sometimes never 

14. These are questions about using your English skills when you are not at 
work. 

•LAST YEAR, before I started class, I UNDERSTOOD WHAT I 

READ in English (for example, newspapers, books, instructions, children's 
school reports, medical and other forms) 

always usually sometimes never 

• LAST YEAR, before I started class, ! SPOKE in English (for example, 
to doctors, store clerks, neighbors.) 

always usually sometimes never 
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• LAST YEAR, before i started class, I WROTE in English (for example, 
medical and other forms, notes to my children's school.) 

always usually sometimes never 



• LAST YEAR, before I started class, I UNDERSTOOD what people 
said to me in English (for example, doctors, nurses, store clerks, children's 
teachers, neighbors.) 

always usually sometimes never 



•NOW I UNDERSTAND WHAT I READ in English (for example, 
newspapers, books, instructions, children's school reports, medical and other 
forms) 

always usually sometimes never 



•NOWISPEAKin English (for example, to doctors, store clerks, 
neighbors.) 

always usually sometimes never 



• NO W I WRITE in English (for example, medical and other forms, notes 
to my children's school.) 

always usually sometimes never 



• NOW I UNDERSTAND what people say to me in English (for 
example, doctors, nurses, store clerks, children's teachers, neighbors.) 

always usually sometimes never 



15. Please think about whether your English skills have changed since you 
started class, and circle the answer that is true for you. 

•NOW I UNDERSTAND WHAT I READ in English. 

a lot better better no change worse 



•NOW ISPEAK English. 

a lot better better no change worse 
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•NOW I WRITE in English 



a lot better 



better 



no change 



worse 



• NO W I UNDERSTAND spoken English. 



a lot better 



better 



no change 



worse 



16. is there anything else you would like to say about your ESL class?. 



17. Do you plan to continue your class? 



If yes, why? Please draw a circle around all that are true for you. 

-- It helps me to talk with my manager 
-- I want to get into a Bull training program 
-- I enjoy learning 
-- I need it for my job 
-- I want to go to college 

-- Other 



If no, why not? Please draw a circle around all that are true for you. 

-- It is too demanding 

- I don't like school 

-- I'm too tired at the end of my shift 

-- 1 have too many things on my mind 

-- Other 



-- Yes 



No 
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Tabulation Bull Student Survey, Spring 1993 

N =20 
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math 
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speak 
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other: 


pronunciations.. .6 hrs./week 







computer class.. .pronunciation. ..speaking better English...! like 
newspaper articles 
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Q.I 2 Always Usually Sometimes Never 

13 6 1 0 

7 9 3 1 

8 6 5 0 
11 3 6 0 



Q.I 3 Last 
year/Now 
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Q.I 5 lot better better no change worse 

5 14 1 0 

3 15 2 0 

0 20 0 0 

5 13 2 0 



Q.16 

COMMENTS: ! would like to learn about speaking reading and conversation in my 
ESL class...! would like to learn ESL in class. I like to speak English well, and future in my 
job hope much better. I would like to be in college sometime... The best program for Bull 
employees.. .This is the best program for all companies in the country... 



Q.I 7 Yes: 20 No: 0 

a 14 

b 13 

C 14 

d 17 

e 9 

Other: 0 
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Questions for Bull ESL Program Student Focus Groups 
July 1993 



Introduction 

Thank you for participating in this focus group. 

1 will ask you some questions about your class. These questions are a follow up 
to the survey you took a couple of weeks ago. We will use the information to 
I improve the program. 

j Please understand two important things: 

I 1 . The questions I ask are not about your teachers. Not about Sue or Mel or 

Annie or Jim. Not about them as teachers. We are interested in your ideas 

I about the program. We need to know what helps you to learn and what doesn't 

help you. 

2. What you say is confidential. Noone will know what you say except Sue and 
me. We will take notes but we will not put your name on them. 

' I. Questions about work-related learning 

I • Are there alot of lessons about work? 

I Is this good or not? Why? 

I • Do you think that coming to class is important to keep your job? Why/what are 

the connections between coming to class and keeping your job? 

• What is the best way for you to learn about wofk-related things? 
I Probes: 

' -- Use work related materials in class? (example: MSDSs in class?) 

I -- Talk to each other alot about what you have to do on your job? 

I Why is this a good way to learn? 

-- Read more Why? 

--Write more Why? 

" Use the computer more Why? 

• Option: if class did not focus on work, what should it focus on? 

I How would you transfer what you learned to doing your job better 
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• Option: What helped you the most to improve your English skills 
Probes: 

- something the teacher did 

- something you learned 

- something you felt about yourself 

II. Using computers 

• Did you use a computer? What did you do on it? 

• How did you like using a computer? Is there anyone here who did not like 
using one? Why didn't you like it? (something in the technology? in the 
instruction? in you?) 

• Did you have alot of support for using computers? Was having support 
important to you? Describe the kind of support you need. 

• How did you feel about: Using the keyboard (typing)? Sitting in front of the 
computer for long periods of time? Reading instructions on the screen that told 
you what to do next? 

Probe: 
scary 

comfortable 

positive 

negative 

If you were uncomfortable, what would make you more comfortable? 

• Would you like to spend more time with the computer? If yes, what would you 
like to learn to do with it? 

III. Tell us how you have changed as a result of being in the class, 
how you would like to change more and how the class can support 
you best to change. 
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Bull ESL Program 



Survey for Managers 1993 



Introduction: 



Thank you for agreeing to participate in the evaluation of the Bull English 
as a Second Language (ESL) program. Your answers to this questionnaire are 
part of a larger evaluation that will help the planning and evaluation to improve 
the program. 

Although we ask you for ratings on individuals, the information gathered 
will be used to guide program improvement. We are interested in what the 
aggregate ratings tell us about the program. Some of the items included in this 
questionnaire have not been covered in class. 



Instructions: 



• Please rate the following communication behaviors for each associate in your 
line or department who is enrolled in the ESL class. 

• Please rate each associate twice. The first time, circle the answer which best 
describes the associate LAST YEAR, before s/he attended the ESL class. The 
second time, circle the answer which best describes the associate NOW. Use 
the comments section at the bottom of each page to explain your answers 
further, if you like. If you need more space, use the back of the questionaire. 

• All the information is confidential. Only the person who handles the data will 
see your name and the name of the associate. 



Your name: 

Name of the associate you are rating: 



The associate: 

1. Speaks up in meetings. 
•LAST YEAR 

always usually sometimes never does not apply 

•NOW 

always usually sometimes never does not apply 
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2. Appears to say what*s really on his/her mind in meetings. 

sometimes never does not apply 
never does not apply 



•LAST YEAR 

always usually 



•NOW 

always 



usually 



sometimes 



3 . Talks informally with leaders. 

•LAST YEAR 

always usually 



•NOW 

always 



usually 



sometimes never does not apply 



sometimes never does not apply 



4. Talks informally with co-workers in English. 
•LAST YEAR 



always 

•NOW 

always 



usually 
usually 



sometimes 
sometimes 



never does not apply 
never does not apply 



5. Asks leaders questions when s/he needs to. 

sometimes 



•LAST YEAR 

always usually 



•NOW 

always usually 



sometimes 



never 



never 



6. Asks co-workers questions when s/he needs to. 



•LAST YEAR 

always usually 



sometimes never 



•NOW 

always 



usually 



sometimes 



never 



does not apply 
does not apply 



does not apply 
does not apply 



Comments: (Please indicate the number of the question you are commenting on) 
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7. Is understandable. 



•LAST YEAR 

always usually 



•NOW 

always 



usually 



sometimes 



sometimes 



never does not apply 



never does not apply 



8. Uses technical terms. 

•LAST YEAR 

always usually 



•NOW 

always 



usually 



sometimes 



sometimes 



never does not apply 



never does not apply 



9. Makes suggestions about how to improve the way we work. 

sometimes never does not apply 
usually sometimes never does not apply 



•LAST YEAR 

always usually 



•NOW 

always 



10. Attempts to read or reads items such as written process changes, 
S.O.D.s, M.S.D.S s, company policies, and blueprints. 



•LAST YEAR 

always usually 



•NOW 

always 



usually 



sometimes 



sometimes 



never does not apply 
never does not apply 



11. Appears to understand items such as written process changes, 
S.O.D.s, M.S.D.S s, company policies, and blueprints. 



•LAST YEAR 

always usually 



•NOW 

always 



sometimes 



never does not apply 



usually soitfetimes never does not apply 

Comments: (Please indicate the number of the question you are commenting on) 
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12, Makes appropriate use of items sucli as written process changes, 
S.O.D.s, M.S.D.S s, company policies, and blueprints. 



•LAST YEAR 

always usually 



•NOW 

always 



usually 



sometimes 



sometimes 



never does not apply 
never does not apply 



13. Appears to understand safety signs. 

sometimes 



•LAST YEAR 

always usually 



never does not apply 



•NOW 

always usually 



sometimes never does not apply 



14, Writes memos. 
•LAST YEAR 



always 

•NOW 

always 



usually 
usually 



sometimes never does not apply 



sometimes never does not apply 



1 5. Writes notes to co-workers (such as a note to the next shift). 

never does not apply 



•LAST YEAR 

always usually 



sometimes 



•NOW 

always usually' sometimes never does not apply 

16. Appears to understand me when I speak. 
•LAST YEAR 

always usually sometimes never does not apply 

•NOW 

always usually sometimes never does not apply 

Comments: (Please indicate the number of the question you are commenting on) 
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17. Appears to understand others (such as engineers) when they speak. 
•LAST YEAR 

always usually sometimes never does not apply 

•NOW 

always usually sometimes never does not apply 

18. Appears to understand co-workers when they speak in English. 
•LAST YEAR 

always usually sometimes never does not apply 

•NOW 

always usually sometimes never does not apply 

19. Appears confident in her/his ability to learn. 
•LAST YEAR 

always usually sometimes never does not apply 

•NOW 

always usually sometimes never does not apply 

20. Appears motivated to continue learning. 
•LAST YEAR 

always usually sometimes never does not apply 

•NOW 

always usually sometimes never does not apply 

Comments: (Please indicate the number of the question you are commenting on) 



21. Is there anything else you would like to say about: 

• How associates have changed? 



• The program?. 
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EVALUATION RESEARCH 

130 Warren Street 
Newton Center, MA 02159 
Phone: 617/527-6081 

TO: Members of the Bull Basic Skills Planning and Evaluation Team 

FROM: Laura Sperazi 

RE: Questions for Managers' Focus Groups 

DATE: September 27, 1993 



There will be two managers' focus groups. One group will be held on 
Wednesday, October 6 from 5:00 - 6:00 p.m.; the second will be held on Friday, 
October 8 from 7:00- 8:00 a.m. Managers can attend either group. Jim, 
Sharon, and Laura will staff the groups. Laura will take the lead in asking 
questions; Jim and Sharon will take the lead in providing information about the 
ESL program and how managers can become more involved in it. 

All managers - except L. Magdalenski (sp?) - have associates who are 
enrolled in the ESL program. All managers are expected to attend one of the 
focus groups. The PE Team is supposed to come up with a more-interesting 
name for the groups than "focus groups." The hope is that the more-interesting 
name and a good outreach effort will ensure that all managers attend. 



Part I: Questions for Managers 

Introduction: 

- Thank you for participating in this focus group 

- These questions are a follow up to the survey you took a couple of weeks ago 
(OK if you didn't fill out the sun/ey; we still want to know what you think about the 
ESL program.) 

- Focus group is divided into two sections: asking you some questions about 
what you have observed about associates' participation in the ESL program in 
the last year or so; and giving you some information about the program for this 
coming year 

- We will use the information you give us today to improve the program 

1 . What new skills, behaviors or attitudes have you observed in associates in 
the last year (or so) that you think are the result of participation in the Bull ESL 
program? 
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Part il: Information about the ESL Program 



Themes for Jim and Sharyn to work with: 

1 . The lEP Process 

• What is the lEP process? Describe lEP process as "checkpoints for 
progress" for the mutual benefit of associate and company. Reassure 
managers that associates are not left on their own to sink or swim but that there 
is a mechanism for tracking progress. 

2. How can managers help to support the ESL program? Specific suggestions 
and discussion re: how managers can support the program 

-- Identify associates who need ESL instruction. What are some of the 
indicators that an associate needs ESL instruction? Use handout.to 
support discussion. 
-- Encourage associates to enroll 
Tell teachers and other education staff that someone needs support 
NOTE: The ESL program will allow one hour of planning time each 
week for each teacher. This means that you can expect to see teachers on the 
floor, observing work, talking to associates and generally becoming more 
knowledgeable about Bull manufacturing processes and work culture. The 
teachers are available to you if you questions about the program, an idea you 
want to pass on, information about a particular associates or associates -- 
whatever. One of the responsibilities which the teachers have is to integrate the 
real demands of work into the classroom. They will appreciate any way you can 
help them do that. 

-- Other ways managers can become more involved in the ESL program. 
Use handout to support discussion. 

* Participate in conferences with the with the associate and teacher (as 
part of the lEP process) 

* Review curriculum, critique it, add to it. 

* Be a speaker in a class 

* Volunteer to be a language coach or tutor 

* Learn more about the Jostens computer learning system which will be 
installed in the plant soon, and create a role for yourself which 
supports the use of the system. 
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Probe: Any changes in: 

• Listening skills 

• Speaking skills 

• Computer skills 

• Reading work documents 

2. As you look forward, what expectations do you have for what the Bull ESL 
program can achieve? 

Probe: Think specifically about associates you know who are enrolled in 
the program. 

• What did you expect from associates in the past year 
because they were enrolled in the ESL program? 

• What do you expect now ? 

3. Did you observe associates doing homework this past year? 
Probe: (1) What exactly did you observe? 

• Doing homework on work time 

• Doing homework "in teams" 

• Talking about homework 

• Showing others that they had done their homework 

• Other 

(2) Did/do you have any opinions about these activities? 

4 What kinds of changes in skills, behaviors, attitudes, or other areas - would 
you like to have tracked as part of the ESL assessment process? 

Question #4 leads us into Part 11 of the focus group where Jim and Sharyn 
convey some information about how the program operates and is integrated 
with overall Bull manufacturing and education goals. This information is 
conveyed with the purpose of increasing managers support of and involvement 
in the program. 
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Part il: Information about the ESL Program 



Themes for Jim and Sharyn to work with: 

1 . The lEP Process 

• What is the lEP process? Describe lEP process as "checkpoints for 
progress" for the mutual benefit of associate and company. Reassure 
managers that associates are not left on their own to sink or swim but that there 
is a mechanism for tracking progress. 

2. How can managers help to support the ESL program? Specific suggestions 
and discussion re: how managers can support the program 

-- Identify associates who need ESL instruction. What are some of the 
indicators that an associate needs ESL instruction? Use handout.to 
support discussion. 

-- Encourage associates to enroll 

-- Tell teachers and other education staff that someone needs support 
NOTE: The ESL program will allow one hour of planning time each 
week for each teacher. This means that you can expect to see teachers on the 
floor, observing work, talking to associates and generally becoming more 
knowledgeable about Bull manufacturing processes and work culture. The 
teachers are available to you if you questions about the program, an idea you 
want to pass on, information about a particular associates or associates - 
whatever. One of the responsibilities which the teachers have is to integrate the 
real demands of work into the classroom. They will appreciate any way you can 
help them do that. 

- Other ways managers can become more involved in the ESL program. 
Use handout to support discussion. 

* Participate in conferences with the with the associate and teacher (as 
part of the lEP process) 

* Review curriculum, critique it, add to it. 

* Be a speaker in a class 

* Volunteer to be a language coach or tutor 

* Learn more about the Jostens computer learning system which will be 
installed in the plant soon, and create a role for yourself which 
supports the use of the system. 




lEP Bull Worldwide Information Systems 



Name 

Class 

Teacher 



Overall Goals 

Where you hope to go with your education in the long run 
or where you hope the class will take you this year. 



Work 



Personal 



Objectives 

What you need to do now to move towards your goals. 



1.. 
2.. 
3.. 
4.. 
5. 



This lEP is to be reviewed every 12 weeks by you and your instructor. 
Signature Date 
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lEP Bull Worldwide Information Systems 

Name 



Competencies achieved that relate to specific objectives 

Objective Competency Mastery Indicator Mastery Date 
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Name; 



Date: 



Instructor; 



STUDENT'S INDIVIDUAL EDUCATION PLAN 



Skills 

Oral Communication/Teamwork 

□ ask questions for clarification 
discuss and interpret reading materials 
give directions and instructions 
make requests 

make small talk (social English) 
n make suggestions 
report a problem 

□ 

□ 

□ 

Listening Comprehension 

□ follow and understand oral directions and instruction 
d extract key information 

record key information 

□ 

□ 
□ 



Date/% of Mastery Date/% of Mastery 



Phonics 

□ identify and produce consonant sounds 
ni identify and produce double and triple 

consonant blend sounds 
n identify and produce consonant digraph sounds 
CH identify and produce short vowel sounds 
O produce vowels before r sounds 



A. Yu and M, Green 



Page One of Four 



ERLC 



H5 



88 



Date/% of Mastery Date/% of Mastery 

Phonics 

□ produce long (double) vowel sounds I I 

O produce long vowel sounds with E I I 

O produce irregular double vowel sounds I I 

□ / / 

□ I I 



Pronunciation 

C!1 practice stress, intonation, pitch 

□ 

□ 
□ 



Vocabulary/Spelling 

□ use QuickWords for spelling of sight words 

□ expand word forms using Spellex 

d determine unfamiliar vocabulary through context 

find synonyms and antonyms for familiar words 

d expand vocabulary through the understanding 
of suffixes and prefixes 

□ use a dictionary 

d understand and use vocabulary as related to 
workplace competencies 

□ 

□ 
□ 



Reading 

Read a variety of reading materials and experience 
a variety of activities focusing on: 
identify alphabet 

CI determining the main idea 

d finding the facts 



I A. Yu and M. Green 
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Date/% of Mastery Date/% of Mastery 



Reading 

□ drawing conclusions 
making inferences 
developing vocabulary 

□ summanzing passages, stories, articles 

□ determining fact from opinion 
d analyze characters 

□ 

g 
g 

Writing 

g write letters of the alphabet in print and cursive form 

g wnte correct word order (subject-verb-object) 

C!!l use correct punctuation and capitalization 

ni combine sentences 

d use process writing (draft, edit, revise) 

g understand the use of editing symbols 

g develop writing using a visual (pictures) guide 

g wnte answers to questions to form a story 

(given who, what, when, where, why questions) 
g group sentences into paragraphs 
g write story based on a given topic 
g wnte summanes 
d write instructions and directions 
g use computer for word processing 

g 

g 
g 
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Grammar 

□ simple present 
present continuous 
past tense 

CH regular verbs 

□ irregular verbs 
C!I future tense 

ni parts of speech 

□ nouns 

C3 verbs 

adjectives 
CD adverbs 

review or introduce new grammar as it occurs 
in curriculum 

□ 

□ 

□ 



Date/% of Mastery Date/% of Mastery 
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Name: 

Date: 

Instructor: 



BULL WORKPLACE CURRICULUM 
STUDENTS INDIVIDUAL EDUCATION PLAN 



Date/% of Masterv Comments 



AROUND THE PLANT 



n 
□ 
□ 
□ 
□ 
□ 
□ 
□ 



Understand abbreviations 
Understand and give directions around the plant 
Name plant areas and functions 
Understand stock and shelving s>'stems 
Name components 



SAFETY 

□ 

n 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
□ 
n 



Read safety signs 

Recognize types of emergency equipment 
Recognize types of safety equipment 
Give and understand warnings 
Be aware of hazards in the workplace 
Read chemical hazard warning 
Understand Right to Know law 
Understand short and long term hazards 
Read MSDS for significant items 



HEALTH 

n Name parts of body 
O Name aches and pains 
Bull ESL Prognm: J. Ward, A. Yu, M. Green 
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Datc/% of Mastery Comments 



□ 
□ 
□ 
□ 
□ 
□ 
□ 

n 
□ 
n 

JOBS 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

n 
n 
n 
n 



Call in sick 

Name work-relaled aches and pains 
Explain s\'niptoms (e.g. to a doctor) 
Write out injur>' report 
Fill out health record form 
Understand simple exam procedure 
Understand health care choices 



Name Job title and basic function 

Identify main functions in job 

Answer questions about job and skills necessary 

Highlight skills needed for job 

Explain job to visitors 

Identify changes in job during tenure 

Write free description of work and work day 

Cany out "value-added" analysis of job 

Fill out forms associated with job 

Read documentation necessar\- for job 



QUALITY 

□ 

n 
□ 
n 



Use defects vocabulary 
Describe quality problems related to job 
Complete quality check forms 
Be aware of continuous improvement procedures 

in company 
Discuss changing roles of workers 

Bull ESL Program: J. Ward /V. Yu, M. Greeti 
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Date/% of Master}' Comments 



□ 
□ 

n 
n 



Understand performance re\-iew process: set goals 



COMMUNICATION 



n 
□ 
□ 
n 
n 
n 
n 
n 
n 
n 
n 
n 
n 



Understand instructions 
Give instructions 
Ask for clarification 
Make requests 

Make small talk (social English) 

Learn "Action" verbs: common and Bull-specific 

Make suggestions 

Report a problem 

Write down ideas 

Develop and apply team skills 
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TEAM EVALUATION AT DATATFC INDUSTRTRS : 
A CASE STUDY FROM A WORKPLACE EDUCATION 

PROGRAM 



Prepared 
by 

Paul Jurmo 



One of seven case studies in 
" Team Evaluation : 
Case Studies from Seven Workplace Education Programs," 
a report prepared in June 1994 
by Laura Sperazi and Paul Jurmo 
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DATATEC INDUSTRIES, INC. 



Fairfield, New Jersey 



I. PROGRAM BACKGROUND 

Program history 

Datatec Industries, Inc. is a rapidly-growing manufacturer of 
components for computerized equipment, with eight plants around 
the U,S, (Their products include computer boards for the laser 
scanners used at Toys R Us and other major retailers.) At the main 
plant in Fairfield, New Jersey in late 1990, the human resources 
director had concluded that many of the primarily-Hispanic 
assembly workers could benefit from an ESL course. The director 
investigated several language institutes and had potential 
participants interview instructor candidates. 

Language Training Institute (LTI), based in nearby Englewood 
Cliffs, was hired, and classes were begun with fifteen learners. These 
learners were divided into three groups according to language 
ability. Instruction was "competency-based," focusing on language 
needs identified by the company and participants. 

Program goals 

To keep up with the rapidly-growing demand for its products, 
the company had in recent years instituted a quality program 
emphasizing high quality work by all employees. The ESL program 
was seen as a way of enabling immigrant production workers to 
participate fully in that quality program. They would thereby 
improve productivity, decrease error, use more-complex equipment, 
and communicate and solve problems in teams. The focus of the 
classes has varied periodjcaUy according to learner needs. These 
include oral skills, writing, spelling, public speaking, expression of 
opinion, fast listening, and grammatical accuracy. The instructor 
consciously integrated assertiveness and team-building exercises 
with language instruction. Presentations made by learners at 
company events served as markers of learner progress. 
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11. BACKGROUND ON THE TEAM 



History of the team 

In its first two years, the Datatec program was coordinated 
primarily by LTI director Joy Noren in consultation with Larry 
Tourjee, the vice president of manufacturing. In early 1993 Joy and 
Larry agreed to have the program serve as a site for Paul Jurmo's 
research. At that point, the following planning team was put in 
place: 

Stakeholders represented on the team 

Tosefa Aboleda : Assembler, program participant 

Lucia Izquierdo ! Assembler, production team leader, and program 

participant 
Paul Turmo : outside facilitator 
Toy Noren : education consultant, program coorduiator 
Patti Scharf : Purchasing agent 
Larry Touriee : Vice president of manufacturing 

Larry and Joy selected these members based on the following 
criteria: 

Patti Scharf: Selected because of her demonstrated interest in the 
program from its inception and because of her willingness to 
mentor learners. Also has regular contact with many of the 
Spanish-speaking employees on a day-to-day basis. 

Tosefa Aboleda and Lucia Izquierdo : Selected because (1) their 
language proficiency level would allow them to participate in 
team meetings, (2) tiiey both express their opinions and ideas 
openly, and (3) they interact with a wide range of production 
employees, especially the ESL program participants. 

HI. THE PROCESS WHICH THE TEAM WENT THROUGH 

Phase I : Initial preparations 

To famiUarize himself with the program ~ and vice versa ~ 
the facilitator, Paul Jurmo, first talked with coordinator Joy Noren by 
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phone. She explained the history of the program, focusing in 
particular on what she needed and had in terms of assessment and 
planning mechanisms. (While the company set broad goals, the 
learners "filled in the blanks" in program objectives by bringing in 
problems and interests from their daily work. She wanted to find a 
way to get more input from higher- and supervisory-level 
management in setting course objectives.) 

She also explained the company context, particularly the 
company's shift toward a "quality team" format. 

Based on that input, the facilitator submitted a proposal to Joy 
and Larry. It outlined the history of the NIFL research project and 
what it might produce for the program stakeholders. It also 
described roles which stakeholders might play on the planning team. 

Larry responded enthusiastically to the proposal, and he, Joy, 
and Paul met to discuss the proposal in more depth. Paul also 
observed a class session and talked briefly with the learners. He 
then was led by one of the more advanced learners on a tour of the 
production area where most of the learners worked. 

Based on the information gathered in the above activities, the 
faciUtator wrote an abridged version of his proposal (called an 
"action plan") which Joy Noren circulated among key management 
personnel at the company. She and Larry identified the above- 
identified (See "Background on the Planning and Evaluation Team") 
team members. 

Paul met with Joy to review the background of each team 
member and the roles they would play in the project. Joy noted that, 
until now, the union had not been involved in the education program 
and would therefore likely not be involved in the evaluation project. 

She noted that learners understand the company's reasons for 
setting up the program: enabling them to understand new equipment 
and to become more involved in their work teams. Many had an 
interest in improving those job-related abilities. Many also were 
motivated for persoricu reasons, such as using EngKsh with their 
children. 

She said that, although the company wants clearer 
communication, more "unity" between American and non-American 
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employees, error reduction, and improved production, more work is 
necessary to analyze the specific tasks which learners need to 
perform to meet tiiose larger goals. She hoped to establish clearer 
communication with management representatives, so that she can be 
more clear about what management needs. To date she has relied 
primarily on input from learners, trying to gear the curriculum to 
both the company's broad goals and learners' personal goals. She felt 
that it is difficult,, given the limited time for the course, to respond to 
a broad mix of company and learner goals. 

Joy hoped that the evaluation process would clarify for her 
how to set goals and to measure them in a way which would 
demonstrate tangible results to those who need to support the 
program. 

She until now has set goals for short, 8 -week blocks, changing 
them depending on what's going on in the company at any time. For 
example, "teamwork" was a focal point for some time, and her classes 
read and talked a lot about it. She has not put the resulting learning 
activities together in a systematic curriculum. 

To measure results, she builds in "competency markers." These 
include a task which learners role play; periodic checklists in which 
learners assess progress toward mastery of various competencies; or 
mini-quizzes on topics requested by learners or Larry (in one such 
quiz, learners spelled words taken from the workplace). While such 
assessment tools produce useful data on learner progress, she feels 
that they need to be better organized. 

In late March and early April, Paul interviewed Patty Scharf, 
Josefa Aboieda, and Lucia Izquierdo, the remaining three members of 
the planning team. They discussed the purposes of the program, 
their roles at the company, what the program has accomplished so 
far, and things blocking learners' participation. These interviews 
thereby enabled Paul to better understand the program, while 
allowing him and team members to get to know each other and 
better understand their respective roles in the evaluation. 
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Phase II : Planning the evaluation 



Clarifying program goals and the information to look 
for 

In the first meeting on April 7th, Paul led the team through a 
90-minute discussion of three questions: 

1. What do we mean by the term "evaluation"? 

2. Who will be the audiences for this evaluation? 

3. What types of information do our audiences want? 

Paul recorded members' answers on flipcharts, and later typed 
up minutes summarizing what the team stated. At the end of the 
meeting, it was agreed that Patty, Josef a, and Lucia would talk with 
representatives from the primary audiences to ask them: 

1. In our evaluation report, what kind of information about the 
ESL program should we include? Why do they want that 
information? 

2. What do they think the ESL program's goals should be? 

On April 21st, the team met for the second time, for nearly two 
hours. Paul again served as discussion leader, leading the team 
through the following questions: 

1. What types of information do our audiences want? 

2. For what purpose do they want that information? 

3. What should be the goals of the ESL program? 

4. For the highest-priority goals,what would be evidence of 
success/progress? 

To answer question #1, Patty, Josefa, and Lucia explained that 
they had conducted interviews with eight co-workers (including the 
company president) in the past few weeks. These sources had 
identified questions which they hoped the evaluation would answer 
for them. 

To answer question #2, the team brainstormed a niunber of 
ways the audiences might use tlie information. These included: 
determining retum-on-investment; and deciding whether and at 
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what level to continue the program, how to improve communication 
within the production area, how to improve the education program, 
and how to motivate learners. 

To answer question #3, Paul first summarized what the team 
members had previously told him (in Phase I) they considered to be 
the program's goals. TWs was a lengthy list, divided into major 
categories of "English on the job" and "English off the job." Each 
category, in turn had sub-categories containing specific tasks (e.g., 
participating in team meetings, identifying problems, asking for help, 
reading notices, etc.). He asked team members to review this list, 
revise it as necessary, and identify which goals were of particular 
importance. 

At this point, the discussion became bogged down. This was 
due to a number of factors including (1) some team members were 
not familiar with the language and concepts being used in the 
meetings, and (2) a large number of complex tasks were being 
presented in a short time. Discussion got stuck on particular details 
(e.g., meanings and phrasing of particular terms). The discussion ~ 
and the team's energy ~ groimd to a near halt. 

Joy Noren then introduced another, more concise list of 
"competencies" which she had been organizing the program around. 
It was agreed that this list was similar to Paul's list and ~ for 
clarity's sake ~ might be a better list to consider as the program's 
goals. 

To try to rejuvenate the discussion, Paul moved on to the last 
question of "For the highest priority goals, what would be evidence of 
success/progress?" However, the team had said that all of the goals 
or competencies discussed so far were important. As time ran out at 
the end of the meeting, it was agreed that team members would on 
their own brainstorm a list of possible indicators (evidence) of 
progress toward goals which they feel are particularly important. 

On May 5th, the team met to identify "indicators" for the goals 
which the team had listed on April 21st. Team members had tried to 
come up with questions to ask related to those goals, and in this 
meeting they shared what they had come up with. 

Unfortunately, the discussion rapidly became bogged down. 
Members weren't clear exactly what diey were being asked to do. 
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They focused on details of wording of a small number of indicators 
and never got to other ones. 

One team member suggested that, rather than develop 
indicators in isolation from the instruments they were to be used in, 
the team should focus on developmg one sample instrument. Time ' 
ran out before anything tangible was accomplished, and team 
members left the room frustrated and confused. 

At this point the facilitator felt he was in danger of "losing" the 
team. He concluded that the team had gotten off to a good start, but 
that in the last two meetings members had become bogged down in 
details of a complex, unfamiliar, and abstract task. In consultation 
with the coordinator, he decided to leap ahead a few steps and do 
some "detail work" which, he hoped, would break this log-jam. 

Designing a strategy for gathering data 

On May 12 th, the facilitator shifted the discussion to deciding 
what data-gathering activities the team might use. He hoped that 
this would give the team something more clear and concrete to focus 
on. 

He presented a list of possible data-gathering activities 
developed by the coordinator. He then revised that list to include 
the following possible activities: 

Gathering information from learners : 

• Interviews with learners 

• Oral simulations (tests) 

• Written simulations (tests) 

• Grammar mini-quizzes 

• Mini-quizzes on workplace vocabulary 

• Observation of learners using particular skills in the workplace 

• Leadership projects (in which learners use particular skills to 
achieve goals set by learners, teacher, and supervisors) 

Gathering informa tion from co-workers and supervisors ! 

• Interviews conducted pre- and post, to identify needs, learner 
abilities, and steps to help achieve goals. 



102 

7 



The facilitator suggested that, to keep the process moving, he 
and the coordinator develop several of the above activities and 
present them to the team at the next meeting. The team agreed. 

Desiging data-gathering instruments 

In the next two weeks, the faciUtator and coordinator divided 
up this task. The coordinator put together most of the above 
activities for gathering information from learners. She planned to 
organize the resulting data in individual portfolios for each learner. 
Results of those activities could then be summarized and presented 
to management. The facilitator, meanwhile, designed interview 
guides for interviews to be conducted with learners and co-workers. 
He also developed an "interest inventory" in which learners would 
identify topics which they would hke to focus on in the classes. 

At a meeting on June 2nd, the team reviewed the draft 
instnmients designed by the coordinator and facilitator. It was 
agreed that the coordinator would try out her instruments in the 
coming month, while the facilitator would interview learners. A 
third team member, Patty Scharf, was to interview several co- 
workers. 

The team also set some guidelines for the interviews. These 
included confidentiaUty, sensitivity, and limiting interviews to 15-20 
minutes each. 

Phase III : Gathering, organizing, analyzing, and 
reporting the information 

Gathering and organizing the da^a 

During the next three weeks, the agreed-upon interviews were 
conducted with learners and co-workers, and in-class assessment 
activities were carried out. Those conducting these activities 
recorded the results in draft form. 



Analyzing the data 

On June 30th, the team met to review tiie data gathered in the 
above activities. When the member who interviewed co-workers 
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reported her findings, her data indicated that the three co-workers 
were generally skeptical about the program and the seriousness of 
many of the participants. 

This finding came as a blow to other team members, who 
questioned not only the validity of those co-workers' observations 
but the tone in which they were conveyed. The team was also told 
that a higher-level manager was not pleased with the direction 
which the ev«iluation had taken. He said that he had expected the 
evaluation to generate quantifiable evidence of learner competencies 
and not get off into feedback from co-workers and questions related 
to how to improve the program. 

This negative feedback from supervisors about the program 
and from the higher-level manager about the evaluation was a shock 
to the coordinator and facilitator. It was agreed that the evaluation 
had not produced clear evidence of learner progress. Instead, a lot of 
attention was being given to negative comments of a small number 
of co-workers who in some cases had litde direct knowledge of what 
was gomg on in the classes. It was agreed that the feedback from co- 
workers led to a number of other questions: 

• Were learners actually learning anything in the classroom? 

• If they were developing competencies, why weren't co-workers 
observing the learners using those competencies on the job? 
Were they simply not paying attention? Or were learners not in 
fact using them? 

• If learners were developing competencies in the classroom but 
not using them on the job, what was preventing them from doing 
so? 

• What can be done to help the coordinator be sure that the 
program focuses on job tasks which the company holds as 
important? 

• What are effective ways of documenting what learners are 
actually learning and then showing that evidence to co-workers 
and others in the company? 

Given the mood and confusion created by the feedback from 
co-workers and from the higher-level manager, die team did not go 
much further in interpreting die data. It instead agreed that, in 
order to show higher management what the team has been doing, the 
facilitator would now take the information gathered in the learner 
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and co-worker interviews and summarize it for interpretation by the 
team at a later meeting. 

The facilitator then spent a good part of a day at the word 
processor, preparing a preliminary summary of the data gathered in 
the interviews with six learners and three co-workers. This 
summary was broken down into three sections; 

1. What we hoped to learn : The six key questions which the 
evaluation set out to answer: 

1. In what ways do Datatec employees have a problem 
communicating in English? How do these problems impact the 
employees and the company? 

2. What should the English program be trying to accomplish? 

3. What in fact is being achieved by the program? Are Datatec 
employees now better able to communicate in English? 

4. What factors are contributing to the program's success? 
What factors are inhibiting progress? 

5. Should the program continue? 

6. If the program continues, what steps need to be taken ~ and 
by whom - to help achieve the program's goals? 

2. How we gathered information : A surmnary of the various data- 
gathering activities used: interviews with learners and co- 
workers, formal and informal testing, the facilitator's earlier 
interviews with team members, and the facilitator's tour of the 
worksite and review of documents. 

3. A ,'>v^mmarv of the information gathered : This section was 
organized around the six key questions listed under "What we 
hoped to learn" above. For each question, the pertinent data 
gathered from the various sources were summarized. At the end 
of each set of data, the question "What is your response to the 
above information?" was presented, followed by lines where the 
reader could jot comments. 
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The facilitator sent copies of this 8-page summary to each team 
member with a memo asking each team member to review the 
summary before the next team meeting. Members had one week to 
look over the data and jot down any comments or questions 
provoked by the data. 

The facilitator and coordinator were sensitive that, at the last 
team meeting, they had received negative feedback about the pace 
and direction of the evaluation. Both the facilitator and coordinator 
had after that meeting sent notes to upper management to respond 
to those concerns. The coordinator and facilitator also met to review 
the preliminary summary and agree on the agenda for the next 
meeting. They agreed that the meeting needed to be well run and 
produce clear results. 

At that next meeting, on August 11th, the hoped-for results did 
occur. The facilitator and coordinator co-facilitated the meeting and 
led the other members through a systematic analysis of each section 
of the data. 

As members responded to the data, their comments were 
recorded on flipcharts. It was agreed that the facilitator would now 
incorporate their comments and the original data into a draft of a 
final report. This meeting ended on a higher note than some of those 
which more immediately preceded it. Members felt they had 
actually generated some useful information and been able to learn 
something from it. 

When writing up the team's analysis into a draft report, the 
facilitator realized that he had additional interpretations of his own. 
He however wanted to avoid mixing his interpretations in with the 
other team members', in order to preserve the integrity of what the 
team members had said. His solution was to add another category 
for each section of the data titled "Outside facilitator's interpretations 
and recommendations." 

In die summary, then, the data were organized according to the 
six key questions which the evaluation had set out to answer. 
Following each set of data were two sets of interpretations: those of 
the on-site team and those of the outside facilitator. 

The facilitator circulated this draft of the final report to the 
team to review prior to the next meeting, held on September 22nd. 



Just prior to that meeting, the facilitator conducted three additional 
interviews of co-workers. The team had asked him to do so because 
they wanted to clarify whether all co-workers felt so negatively 
about the project as did the first three who were interviewed. In the 
three additional interviews, the facilitator found a good deal of 
support for the program and for the learners. 

At the team meeting which followed, the team reviewed the 
contents of the draft report. Members agreed with the contents of 
the report and asked that the facilitator also prepare an executive 
summary. That summary was to emphasize concrete actions which 
might be taken - and which had already been taken ~ in response 
to the evaluation process. The team felt that the upper-level 
managers who constituted the primary audience for the report would 
be particularly interested in evidence of how the program was 
continually improving itself. 

The facilitator was pleased to see that, in this September 22nd 
meeting, the team had suddenly taken control of the process. The 
discussion flowed smoothly and naturally, with a positive, action- 
oriented tone. In meetings up to that point, the facilitator had stood 
at the helm, directing the content and flow of discussions. In this 
meeting, team members didn't look to him for guidance but instead 
spoke freely and pointedly about how to pull together a final report 
and what directions the team wanted to follow next. 

Preparing a strategy for reporting the findings 

Thus, in the September 22nd meeting, it was agreed that the 
facilitator would add an executive summary to the report which 
emphasized concrete, positive actions the team had been taking to 
improve the program based on evidence gathered in the evaluation. 
In other words, the executive summary was to avoid dwelling 
unnecessarily on negative comments, but instead emphasize positive 
action. 

To help the facihtator prepare this action-oriented executive 
summary, the coordinator provided various memos and other 
documents showing examples of the actions she had been taking in 
recent months to improve the program. The facilitator incorporated 
those examples with other recommendations which the team had 
come up with in the analysis process. 
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He faxed the resulting report to team members for additional 
feedback. By October 20th, the final version of the report was in the 
mail to the team members. 



At the September 22nd meeting, in addition to making final 
decisions about what went into the final report, the team decided 
what it would do with the report when it was done. The team 
decided to prepare a summary of the key findings for the company 
newsletter. It was also agreed that copies of the report would be 
submitted to the upper-level managers who were to have been the 
primary audience. 

Team members also agreed that, as the report was prepared, 
evaluation teams need to take special care to maintain confidentiality 
and sensitivity to the feelings of individuals affected by the 
evaluation. For example, if an informant is critical of someone else's 
performance, such criticism needs to be handled with sensitivity, to 
avoid embarrassing anyone involved or creating unnecessary conflict 
or hard feelings. 

Reporting the findings 

Key findings from the report were summarized in an article in 
the company newsletter. This summary emphasized the progress 
learners had made and its impact on the production area. 

The report was also submitted to the plant managers who 
constituted the primary audience for the evaluation. At this writing, 
however, it is not known what response the managers had to the 
report. 



Phase rV : Deciding what happens next 
Taking follow-up action 

As one result of the evaluation, the coordinator began 
developing ways of involving co-workers more actively in the 
program. For example, co-workers were invited to observe classes, 
and one agreed to do so. Team members also prepared a formal 
invitation for co-workers at all company sites to become language 
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mentors to class participants and thereby involve themselves in 
direct, useful ways on an ongoing basis. 

Within a month a newly-reinvigorated mentorship program 
was in place. Co-workers at not just the New Jersey plant but in 
other company facilities around the country have agreed to 
communicate with learners by phone and in writing. At the New 
Jersey plant, co-workers who signed up as mentors agreed to meet 
periodically with learners. A list of volunteers was circulated to 
learners, and learners chose whom they wanted to work with by 
"signing up." Learners then took the initiative to contact their 
mentors to get started. Mentors will be "circulated" every two 
months, to help co-workers and learners to get to know each other 
better and break down traditional barriers between learners and co- 
workers. 

The team itself might be going through a transition. One 
supervisor has shown an interest in joining the team (which 
members renamed the education "action team.") The coordinator 
has suggested vaiious ways of getting more employees involved in 
the team, including rotating members or holding open sessions which 
anyone can attend. 

In her communications with supervisors and upper . 
management, the coordinator felt that they had a general sense of 
progress about the class. She also talked individually with the upper 
manager on the team about how to link the program more directly 
with "teamwork" initiatives. This was an issue uncovered in the 
evaluation. One teamwork activity will be a "bilingual teamwork 
day" with special activities aimed at building communication 
between language groups. 

Evaluating the evaluation 

After the final report had been submitted to the team, the 
facilitator asked team members whether they might give feedback ~ 
either in a team meeting or in writing ~ on the evaluation process 
itself. 

Feedback from coordinator : In writing, the coordinator gave the 
following feedback: 



1. "The team concept witii stakeholders from all dimensions of the 
company is a sound one. Unfortunately, the team frequently 
bogged down in negativity and confusion which I think could be 
avoided to a large degree in the future by: 

• "Brief, practical tasks designed for information gathering (e.g., 
using simple charts where things could be listed or filled in, 
short lists of direct questions. A lot can be accompUshed in a 
short time if the task is very focused. 

• "Brief explantation of tasks with a minimum of academic 
language to allow 'easy entry.' 

• "Balance of time early on in the process between gathering 
feedback from stakeholders and planning for change in 
response to feedback. (Practical results of the team's time need 
to be evident sooner.) 

♦ "Spend time up-front with some team-building exercises to 
assure that the lines of communication are as honest as 
possible and to build trust in the group. 

♦ "Invite co-workers to attend an open team meeting so they 
can better understand what the team is about and how it could 
benefit them." 

2. "My original hopes for the project (. . . and what I actually 
achieved): 

♦ "(Hoped for) better communication with stakeholders at 
Datatec. (Achieved) some progress in improving 
communication. 

♦ "(Hoped for) assistance from evaluation consultants in 
developing better assessment tools and attitude inventories 
which would strengthen the program. (Achieved) not what I 
had anticipated. 

• "(Hoped for) some national/state attention for Language 
Training Institute. (Achieved) nothing as yet. 
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• "(Hoped for) a knowledge of the team process as it applies to 
workplace education. (Achieved) I learned a lot through our 
difficulties as well as through the positives. 



3. "What I liked and didn't like: 
Liked : 

• A chance to reflect on the program with others. 

• A team model based on representation of all stakeholders. 

• Constructive feedback and suggestions which contributed 
toward the improvement of the program. 

• Documentation of our project. 

Didn't like : 

• The nature of activities in the meeting rarely motivated 
people to attend meetings and generally did not spark full 
participation or positive energy. 

• The team never formed a true team. 

• The project did not adequately compensate Datatec 
employees or the education consultant for time required on the 
project outside of work or class time. 

• The data-gathering process appeared to be abstract and so 
lengthy that the credibility of the team effort became an issue 
among Datatec management. I felt responsible because I had 
introduced the project to Datatec. 

• Negative destructive criticism was counter-productive. 
(Negative constructive criticism was useful.) 

Feedback from Purchasing Agent : 

• "Although, in recent months, upper management has a 
greater recognition of the ESL program's progress, I do not feel 
that it is largely due to the evaluation. Again, I don't feel there 
is a great deal of quantitative information in the report to give 
managers this impression. Instead I feel that it is due to the 
ESL students taking on a much more active co-worker role in 
the company. This may be due to the Mentor Program, which 
was a result of our study. Also, recently production employees 
have volunteered for more extra-curricular activities, and may 
be a result of more confidence in speaking English. 
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• "Honestly, I feel the study did not accomplish its original goal, 
which I understood was to identify marked improvement in 
the students' performance as a result of their new-found 
English skills. However, it did give a starting point to identify 
progress in the future. By marking their current levels of 
English proficiency, we will better be able to identify their 
progress in the future. Joy and I also meet regularly to 
brainstorm on new ideas to keep the program more interesting 
and providing short term goals for the classes to reach. 

• "Another thing I felt to be misrepresented in the evaluation 
was the role of the facilitator. I felt that your role was more of 
an observer, who gave Uttle direction in meetings and should 
accept more responsibility for discussions becoming bogged 
down. One of the roles of facilitator is to keep meetings moving 
smoothly and discussions from getting off track. 

• "I did distribute our report back in the fall and, 
unfortunately, received no response from managers or co- 
workers. I don't feel discouraged, though. It has been an 
extremely busy year, and I am confident that the report was 
reviewed. 

• "Again, I feel that the outcome of the committee's charter 
was positive. Joy has developed several quantitative 
measurement systems, the Mentor Program is going 
exceptionally well, Larry is more confident in the level of 
proficiency of most participants, and it has really opened doors 
of communication between Joy and other Datatec employees to 
receive feedback and suggestions for the program's goals." 

Other team members gave similar indirect feedback through 
the coordinator. Some apparently were confused by the technical, 
"academic" nature of the process. Some also were confused and 
turned off when they didn't see any tangible results for their efforts. 
(For example, meetings would end in confusion; there would be time 
lags between meetings; it was not clear what response the intended 
audience had to the team's final report.) 
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IV. REFLECTIONS 



Paul Jurmo's comments: 

• The team members' analyses of the team process (See 
"Evaluation the evaluation" above) were essentially on target. The 
tasks given the team too often did become bogged down in 
confusion, leading to decreased team member interest. 

It might be noted, however, that essentially the same activities 
were used in other sites, with more-positive results. Why the 
difference between the Datatec site and other sites? 

Time 

In my view, a key difference at Datatec was the 
relatively small size of the company and education program. 
The small nimibers of people involved meant that only a small 
number of people could serve on the team. Team members in 
turn had limited time they could spare away from their jobs, 
making it difficult to set aside enough time to consider all the 
questions being presented by the facilitator. 

In other sites there were more people to spread the work 
around. At Datatec, there were only a few people and they 
could spare only a limited amount of time for a project brought 
in from the outside. 

Motivation 

The evaluation team concept at Datatec was an idea 
brought in by an outsider. In other sites, stakeholders had 
already formed some kind of collaborative planning structure 
and come to the conclusion that some kind of evaluation was 
needed. The Datatec program was probably to a certain degree 
getting into evaluation as a courtesy to the outside facilitator. 
When the process became bogged down, team members 
naturally questioned whether it was worth continuing if the 
reward for the effort was to go primarily to an outsider being 
paid for this project rather than to team members themselves. 

It also appeared that some team members were looking 
primarily for quantitative evidence of positive program impact 
while the facilitator was hoping to provide an opportunity for 
team members to develop their own summative and formative 
data and a system for continually collecting and using such 
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data. In other words, those team members had a different 
expectation for the evaluation than did the facilitator. 

Infrastructure 

The Datatec team was pulled together to work on this 
research project. It was not something which already existed, 
and team members did not have a team identity and a history 
of working together as a decision-making unit. Extra time was 
probably needed to create and nurture such a team 
infrastructure, but as noted above -- that time didn't exist. 
The faciHtator tried to get on with getting the group to perform 
tasks which they weren't fully prepared for. This contributed 
to confusion and miscommunication. 



The facilitator didn't realize tiiat the Datatec team faced these 
limitations. He instead tried to forge ahead, to test his model. He 
didn't understand tiie causes of tiie confusion which resulted and 
didn't know how to handle tiiat confusion. If he had, he might 
have done the things which tiie coordinator and purchasing agent 
suggested: clearer explanation of tasks, simplification of tasks, 
exdra team-building activities, keeping meetings moving and 
discussions from getting off track, and so forth. 

Despite tiiese bumps in tiie road, the process did produce the 
positive results cited by tiie coordinator and purchasing agent 
under "Evaluating tiie evaluation" above. The outcomes which 
team members seemed to value were: clearer measures of learner 
abilities; improved communication between tiie coordinator and 
otiier company employees; greater involvement of other 
employees in tiie program (e.g., as mentors, shapers of curriculum, 
etc.); an increased willingness of learners to take on more-active 
roles in tiie company; and an overaU greater interest by company 
employees to pay attention to tiie program, monitor its progress, 
and help it to work. These are potential positive outcomes which 
otiier evaluation teams might aim for. 

Otiier teams, however, should carefully assess team members' 
time, skills, and interest level before embarking on an evaluation. 
Teams should also be clear what tiie facilitator's and otiier team * 
member's roles will be. More specifically, in a traditional 
"outside" evaluation, most of the responsibility for designing 
instiruments, collecting information, and analyzing it is in the 
hands of the outside evaluator. In a team evaluation, the 
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facilitator helps teams make decisions about what iiifomiation 
they want, develop their own instruments, gather data, and 
analyze the data. Teams need to be clear who plays what role in 
the evaluation process, given the time, skills, and interest level of 
all team members. 

Teams can then agree on tasks which each team member can 
play. These tasks should be do-able, feasible, and meaningful for 
those involved. Members need to feel success ~ a sense that they 
personally and the team as a whole are accomplishing something 
useful ~ or they will get discouraged. 

Teams also need a mechanism for dealing with confusion, 
frustrations, or conflicts which arise, so that they don't irndtirmine 
the team's efforts. For example, if a team member agrees to 
design an instrument and doesn't do so, or if a facihtator tries to 
accompUsh too much in an hour-long meeting, team members 
should have t he trust and opportxmity to resolve those problems 
promptly ~ in the spirit of constructive criticism and continuous 
improvement. 
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EVALUATION REPORT ON 
DATATEC EDUCATION PROGRAM 



FXFCITTIVE SUMMARY 

In early 1993, Datatec organized a team to evaluate the 
company's employee English-as-a-second-languge program. The 
team was composed of four Datatec employees, the outside education 
specialist who runs the program, and a researcher funded by the 
National Institute for Literacy. From spring until September, team 
members collected information from program participants, several 
co-workers, and other sources. Summarized below are the team's 
findings and recommendations, organized aroimd aroimd six research 
questions: 

Ouestion#l : In what ways do Datatec employees have a 
problem communicating in English? How do these problems 
impact the employees and the company? 

Oral communication is particularly important at Datatec. 
Employees are expected to be able to not only understand 
information conveyed from others but to make suggestions and give 
feedback to ensure product quality. Those who don't speak English 
as their first language too often have problems communicating in 
English. While many of these employees are able to understand 
what others say, they are reluctant to try to express themselves in 
English out of "shyness" (embarrassment, fear of making a mistake). 

When employees can't or won't use English with co-workers, 
not only do communications slow down but team morale can be 
jeopardized if other co-workers become resentful. 

In addition to oral English, all employees should be able to read 
and write basic documentation used in the plant. 

Question #2 : What should the EngUsh program be trying to 
accomplish? 

From the beginning of the program, stakeholders have 
consistently stated that the program should aim at strengthening the 
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abilities of participating employees to use EngUsh to solve problems 
they face on and off the job. More recently, the planning team has 
identified a second goal for the program: to help the organization as a 
whole to facilitate communication and teamwork among employees, 
particularly given the multicultural nature of the company 
workforce. 

Question #3 : What in fact is being achieved by the 
program? Are Datatec employees now better able to 
communicate in English? 

Most learners appear to be using English on the job and with 
their families more successfully today than they did even one year 
ago. This is supported by anecdotal feedback from co-workers and 
learners, as well as by a number of assessment activities being used 
in the classroom. In the classroom, learners demonstrate growing 
abiUties to use English in situations they encounter on the job. 

In addition to the above hoped-for outcomes, a number of 
xmanticipated benefits have resulted. For example, English-speaking 
co-workers cite improved relations with Spanish-speaking 
employees. 

At the same time, there have been some instances of negative 
impacts. For example, some English-speaking co-workers resent the 
fact that, despite being given free classes, some learners continue to 
speak Spanish at work and/or don't take the classes seriously 
enough. 

Question #4 : What factors are contributing to the program's 
success? What factors are inhibiting progress? 

Positive factors : 

• A dedicated, creative, helpful teacher. 

• A curriculum which is continually revised to keep it relevant as 
new needs and interests emerge. 

• Co-workers who show support for learners by paying attention 
to them, making sure learners understand, and helping them to 
speak correctly. 

• Hardworking learners. 
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• An environment which encourages all employees (including 
learners) to try new things and develop new skills. 



Ne gative factors : 

• Limited time for class. (It's very difficult to learn a new 
language with only 1.5 hours per week of formal instruction.) 

• Other responsibilities (e.g., overtime, family, commuting, other 
jobs) which make it difficult for learners to study outside class. 

• Inadequate commitment and effort by some learners. 

• Lack of clarity (or perhaps interest) among co-workers about 
what is being achieved in class or how they can help. 

• Confusion by co-workers about why learners use Spanish on the 
job even though they are attending class. 

• Learners sometimes feel they aren't rewarded sufficiently even 
if they do improve their language skills. 

• Some learners don't see themselves as ever being "English- 
speakers" or higher-status employees. They thus don't set high 
enough goals for themselves or adequately try to integrate into ~ 
or take on new roles in ~ the Datatec "culture." 

• Co-workers who are insensitive to the special needs of limited- 
English-proficient employees. 

Question #5 : Should the program continue? 

Virtually all respondents said that, yes, the program should 
continue. A few co-workers, however, stressed the need for stricter 
requirements for admission to the course and for being allowed to 
stay with the course. Those who don't demonstrate hard work 
should not be allowed to participate. 

Question #6 : If the program continues, what steps need to 
be taken ~ and by whom ~ to help achieve the program's 
goals? 

The following are key recommendations based on feedback 
from learners and co-workers. Note that many of these 
recommendations are already being implemented, many as a result 
of this evaluation. 

Communicate with others outside the classroom. 

Too often, key Datatec personnel- including both higher-level 
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managers and co-workers on the shop floor ~ aren't aware of 
what is being achieved in the classes or how they might support 
learners' efforts to learn English. The following measures might 
improve commimication among all stakeholders: 

Re gular updates from instructor to management : The 
instructor has already increased the number of progress 
reports and requests for feedback she is submitting to 
company management. This should be continued. In turn, it is 
helpful when management pays attention and responds to 
those comunications, as they are a quick way to ensure that the 
program stays on track and gets the support it needs. 

Ongoing team meetings, with expanded membership: Some 
version of the current planning team should be continued to 
provide an ongoing fonmi for monitoring and fine-tuning the 
program. Membership might be expanded to in some way 
include other n. inagement representatives, co-workers, and 
learners. (Perhaps not all team members need to attend every 
team meeting, but they would be available for input and other 
help as needed.) 

Mentoring program: Team members are currently exploring 
establishing a "language mentor" program in which English- 
speaking co-workers could volunteer to be paired with 
learners. In those pairs, the English-speakers would be 
available to help with particular English-language questions. 
At the same time, the co-workers would better understand the 
needs, interests, and abilities of Spanish-speakers and be more 
likely to support the learners and the program. For this to 
work, special guidelines are being developed - with input 
from learners ~ to show mentors and other employees specific 
things they can do to help learners. Mentors might even 
include employees in other plants who would correspond with 
the learners. 

Open houses : The program might invite co-workers to sit in on 
classes, hear reports from learners, respond to questions from 
learners, etc. 

Newsletter articles : Team members ~ and possibly learners 
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themselves - can write brief articles for the company 
newsletter, highlighting what is being achieved and soliciting 
co-worker ideas and help. One variation of this: interview 
employees and publish humorous excerpts. (One such article is 
currently being written by team members.) 

Class publications : As class exercises, learners might prepare 
action plans, autobiographies, or other writings about topics of 
interest to them. These could be compiled into anthologies and 
distributed within the plant. These would give the learners a 
reason to practice their English, while helping co-workers to 
better understand the learners and the program. 

Research projects : As a class activity, learners are selecting 
Datatec-related topics to "research" via interviews with other 
employees. Learners make class presentations based on the 
responses given in the interviews. Topics covered to date: 
teamwork, stress management, sexual discrimination and 
harassment, and company volunteer activites (e.g., the Cancer 
Foundation). In so doing, learners develop skills and 
knowledge, while getting to know co-workers; co-workers in 
turn get to know the learners and better imderstand what's 
going on in the classes. 

International desserts party : Promote understanding of 
cultural diversity by hosting another party to which employees 
bring ethnic desserts. 

Play ; Learners join with other employees to produce a play, 
possibly to perform at international desserts party. 

"Speak to Me in English" buttons : Learners might invite 
conmiunication with English-speaking co-workers by wearing 
buttons with the words "Speak to Me in Enghsh." 

Mini-Frost Valley : Host a one-day or half-day team-building 
session for learners and others who couldn't attend the Frost 
Valley team-building retreat. 
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Deal explicitly witJi intercultural issues. 

Some co-workers don't understand why it seems to take so 
long for class participants to learn English. Some appear to 
suspect that learners simply aren't trying enough. 

As part of the above communications with otliers outside the 
class, the team might deal with this issue head-on by discussing 
tliis and related issues directly with co-workers. The team might 
meet with co-workers who have questioned the pace of learning 
and ask directiy whether those co-workers feel resentful. If so, 
why? What might be done about it? 

Learners might delegate one or more of their fellows to serve 
as "cultural ambassadors" to the rest of the workforce. At the 
same time, those who volunteer as "language mentors" might play 
a similar role of facilitating communication between learners and 
co-workers. 

" Intercultural issues might be made an explicit focus of Datatec 
training and team-building activities. Witliin the ESL class, 
learners might analyze instances in which cultural differences 
blocked communication and develop strategies for reducing 
intercultural problems within the company. 

In the process of discussing these issues, it might become clear 
that, for learners to reach their full potential at Datatec, the 
company might need to do more than just provide an English 
class, valuable though the class is. The company might also have 
to re-think how communication and teamwork are structured, so 
that learners work in an environment where they are truly able 
to use the skills they are developing in the education program. 
The company might, for example, consider the following 
initiatives: rewriting documents to ensure "clear language;" 
providing management personnel with special training in 
intercultural communications; and restructuring team meetings 
and other "feedback" sys' ms to make it easier for self-conscious, 
limited-English-proficient employees to speak up with the ideas 
they have for improving operations. 
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Clarify expectations and monitor progress. 

The above activities should help all stakeholders clarify 
expectations for the project. Other ways of doing so might include: 

• Establish a portfolio assessment system. The instructor has 
already reorganized learner records in individual "portfolios" 
for each learner. Each portfolio contains vatrious kinds of 
evidence of the learner's goals and progress toward those goals. 

This is helpful for the instructor, the learner, and anyone else 
interested in understanding what is being achieved in the class. 

• Demonstrate results in a clear, quantified, graphic way . The 
instructor is now, where possible, documenting learner 
progress toward goals in charts or other graphic forms familiar 
to Datatec co-workers. The instructor can show, for example, 
how participants have all successfully demonstrated twenty 
oral competencies in real-life situations (e.g., interacting with 
class visitors) and in simulations. 

• Analyze what role communication plays in the plant . The 
team might conduct an organizational needs analysis to clarify 
what activities really require reading, writing, listening, and 
speaking skills. This analysis should also analyze written 
materials used in the plant, to clarify how they might be 
written more clearly to make them more accessible to learners 
and other employees. This would help the team clarify to what 
extent commimication is blocked by learners' English skills or 
by other factors like poorly-run meetings, poorly-written 
documents, etc. 

• With input from learners and other stakeholders, establish a 
set of "ground rules" or guidelines which all learners need to 
abide by. These might include doing homework, regular 
attendance, showing progress, etc. Learners should then 
formally agree to abide by those rules via a learning contract. 
(The instructor has already set up a learning contract with each 
learner, with the specifics of each contract decided by the 
learner.) 
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♦ Establish a fair set of guidelines for clear rommunication and 
tegmwQrk. With input from learners, the team might prepare a 
set of guidelines for using English in the plant. Guidelines 
might emphasize the links between clear communication, 
teamwork, and quality. For example, they might go something 
like this: 

"Datatec emphasizes quality work in a team environment. 
Clear communication is key to making this work. All 
employees should use English to communicate with co- 
workers in a clear way, to identify problems, and jointly 
solve them. Communication and teamwork are two-way 
streets, requiring respect and extra effort to speak and 
write clearly and to listen carefully ..." 

Revise curriculum content. 

As the above activities elicit clearer input from all stakeholders 
about specific knowledge and skills to focus on in the program, the 
instructor should revise the curriculum accordingly. 

One example: the instructor is developing a leadership project 
which would require initiative, assertiveness, goal-setting, and 
interaction with American employees. All stakeholders are being 
asked to give specific ideas for such a project. 

Provide more practice time. 

Learners would like the company to find some way to provide 
more class time. The team should clarify whether this can be 
done in a way which would not be disruptive to production 
schedules. 

At the same time, learners need to recognize that, if they are to 
make much progress, they need to make special efforts to use 
English outside of class hours. They need, for example, to try to 
use English with co-workers on the job. and they need to find 
ways to practice using English (e.g., tapes, TV shows, reading, 
night school, etc.) outside of class, as well. 
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Although most learners admittedly have many responsibilities 
which make practicing outside class difficult, they are unlikely to 
make much progress unless they do so. Learners must decide 
how important learning English is to them and then make special, 
sustained efforts to practice whenever they can. "Practice makes 
perfect" is a simple rule which applies to everyone, no matter 
what we might be trying to learn. 

Provide incentives and opportunities to learners. 

The company should continue to encourage learners to improve 

their abilities and take on new challenges. Learners should be 
encouraged to participate fully in job-rotation, improving 
operations by giving feedback to engineers, and various training 
and team-building activities. They will thus see reasons for 
learning English and using it. 
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INTRODUCTION 

From April through September of 1993, an education planning 
team at Datatec Industries in Fairfield, New Jersey conducted an 
evaluation of the plant's employee English-as-a-second-language 
(ESL) program. The team used a new, collaborative approach to 
evaluation being developed with funding from the National Institute 
for literacy by workplace literacy specialist Paul Jurmo. Rather than 
rely on an outside expert to evaluate them, the program's 
stakeholders would ~ with Paiil Jurmo as facilitator ~ plan and 
carry out an evaluation of their own program. 

Members of Datatec's planning team were: 

• Tosefa Aboleda: Assembler, program participant 

• Lucia Izquierdo : Assembler and production team leader, 
program participant 

• Paul Turmo : Director of literacy Partnerships, outside facilitator 
for the team project 

• Toy Noren : Director of Language Training Institute, ESL 
instructor 

• Patty Scharf : Purchasing agent 

• Larry Touriee : Vice president of manufacturing 

This report presents: 

• What we hoped to learn 

• How we gathered information 

• How we organized and interpreted the information and made 
recommendations 

• The information we gathered and our interpretations and 
recommendations 

This report is presented to the various stakeholders who have 
an interest in the Datatec education program. We hope that it will 
help them clarify how employee education can help both individual 
employees and the larger organization grow. 
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WHAT WE HOPED TO LEARN 

In meetings in April 1993, the Datatec team agreed that it 
would collect several types of information related to the education 
program. We would, in effect, attempt to answer the following 
questions: 

1. In what ways do Datatec employees have a problem 
commimicating in English? How do these problems impact the 
employees and the company? 

2. What should the English program be trying to accomplish? 

3. What in fact is being achieved by the program? Are Datatec 
employees now better able to commimicate in English? 

4. What factors are contributing to the program's success? What 
factors are inhibiting progress? 

5. Should the program continue? 

6. If the program continues, what steps need to be taken ~ and 
by whom - to help achieve the program's goals? 
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HOW WE GATHERED INFORMATION 



Team members gathered information in several ways from a 
nimiber of sources, as follows: 

• Cn-worker interviews : Patty Scharf and Paul Jurmo interviewed 
six English-speaking Datatec employees who work closely with 
participants. 

• Participant interviews : Paul Jurmo conducted individual 
interviews with six program participants, and briefly met with 
learners in their classes. 

• Formal and informal testing: As part of day-to-day instruction, 
Joy Noren measured learner competencies using several informal 
and formal assessment instruments. 

• Interviews with team members: Prior to the above data- 
gathering activities, Paul Jurmo interviewed all members of the 
team to clarify how learners currently use ~ and should use ~ 
the English language on and off the job. 

• Observation of worksite and revie w of documents: Paul Jurmo 
informally toured learners' work areas, observed printed 
materials used at work, and reviewed work-related documents. 



HOW WE ORGANIZED AND INTERPRETED THE 

INFORMATION 
AND MADE RECOMMENDATIONS 

When the above data-gathering activities were completed, the 
outside facilitator identified which information corresponded to 
which of the original six questions the team set out to answer. (See 
"What We Hoped to Learn" above.) For each question, the facilitator 
organized the information according to its source. 

With the data organized in this way, the facilitator presented a 
written summary of the information to tiie other team members. As 
individuals, they took a few days to review it and jot down 
comments or questions provoked by the data. 

The team then met and went through the information, question 
by question. Team members interpreted and made 
recommendations in response to the data. These interpretations and 
recommendations were recorded on flipcharts and then compiled 
into the section directiy below, titied "The Information We Gathered 
and Our Interpretations and Recommendations." 
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THE INFORMATION WE GATHERED 
AND OUR INTERPRETATIONS AND RECOMMENDATIONS 

Summarized below are the information gathered and the 
team's corresponding interpretations and recommendations. The 
outside facilitator has added his own interpretations and 
recommendations in sections so named. 

Qiipstion #1: In what ways do Datatec employees 
have a problem communicating in English? How do 
these problems impact the employees and the 
company? 

w|i^t r^V^ sourcps said; 

Planning team members are most articulate in identifying 
specific tasks requiring good English communication skills. Most 
of these are oral commimications which learners need to perform 
with co-workers on a day-to-day basis. These include asking for 
clarification and help, making suggestions, and generally being 
prepared to handle new equipment and team-decision-maldng 
procedures. 

Although learners and co-workers aren't as specific in their 
responses to this question, they do tend to see "reluctance to 
speak English out of embarrassment" as a common problem 
among learners. That is, both learners and co-workers tend to 
feel that learners "understand" English but fear speaking it 
because they don't want to make mistakes. This blocks smooth 
communication among English-speakers and those who don't 
speak English fiuently. This is especiaUy true in the context of 
team meetings and otiier situations where employees are 
expected to make suggestions. In such cases, learners still tend to 
rely on "interpeters" or keep quiet. 

A review of shop-floor documents indicates that, ideally, all 
employees would be able to fill out forms (e.g., 'Tailed Material 
Report"). However, many employees don't do so, and the company 
appears to "work around" this fact. This suggests that (1) it is not 
very important to collect the information asked for on the form, or 
(2) the information is actually collected in other ways, instead. 
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Similarly, employees are provided with written instructions 
(e.g., Guidelines for the Ragen Assembly), but it is not clear to 
what extent employees really have to read those instructions. 
Again, this suggests that (1) the information on the guidelines is 
not very important, (2) employees already know the information, 
or (3) employees get the information through trial and error or 
oral instructions from co-workers. 

Other forms used appear to require very Httle knowledge of 
English per se. Employees appear to need to be able to copy part 
names and nimibers off another label and write in quantities in 
appropriate boxes. 

Team's responses to the above information : 

• Oral commimication (both listening and speaking) is particularly 
important at Datatec, where much day-to-day commimication is 
done oraUy (i.e., not in written form). This is a need consistently 
identified by learners. 

• But written communication (reading and writing) is also 
important The team should work with the company and learners 
to clariiy where reading and writing are really needed in the 
production process. 

• All commimication skills (speaking, listening, reading, and 
writing) should be taught. New class activities - like class 
presentations ~ will give learners practice in using many 
different skills at once. 

Outside fa cilitator's interpretations and 
recommendations : 

• As is true in many workplace education programs, there 
appears to be a lack of clarity among stakeholders at Datatec 
about specific uses of English which the program should focus on. 
(Most respondents said only that learners need help with 
"shyness" about using English.) Decisions about specific course 
objectives and content seem to be left in the hands of the 
instructor. 

• This is understandable because (1) most employees have never 
been in the position of setting goals for an education program and 
(2) they are very busy with their own responsibilities. 

• It is important, however, for other stakeholders — managers, 
supervisors, co-workers, and learners - to be actively involved in 
program goal-setting and in the monitoring of program progress. 
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• To do this, the planning team might expand its membership to 
include other people who have shown an interest in the program. 
They might meet periodically to give feedback on needs in the 
plant which the program might help with. They might also make 
suggestions about how the program could be improved. 

• Getting other employees involved in these ways needs to be 
done in a cost-effective manner. Rather than ask employees to 
leave their jobs to attend education planning meetings, the team 
might hold monthly limch get-togethers, perhaps with desserts 
provided by learners. This would give people an opportunity to 
commimicate about the program while building support for it 

• Within the classroom, the instructor might use individual 
"portfolios" as a way of getting learners to keep track of their 
progress. Learners can, for example, keep samples of their 
written work, keep track of what they're doing for homework, 
note examples of how they are using English, etc. This will 
encourage learners to be more thoughtful about what they are 
doing in class. 

• The team might also encourage the company to examine how 
they use written and oral English in commimications. For 
example, how clearly do speakers in work meetings articulate 
their speech? Do they really encourage all employees to speak up, 
or are the ESL students assimied to not be able to contribute to 
the discussion? Are memos, instructions, and newsletters written 
clearly, with short sentences free of imnecessary jargon? 

Question #2 : What should the English program be 
trying to accomplish? 

What our sources said : 

Again, it is members of the evaluation team who present the 
most specific examples of competencies which learners need to 
master. 

Learners generally speak in very broad terms of "improving 
communication" and, when probed, give a few scattered examples 
of instances when they might like to better use English on and off 
the job. 
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Co-workers have little to say on this question of program goals. 
They seem to feel left out of this discussion of what the program 
should be achieving. 

Team's responses to the above in formation: 

• The team was siuprised ~ and a bit discouraged ~ that some 
co-workers didn't see the progress being made by learners. On 
the floor, co-workers frequently commend and encourage learners 
for their efforts to learn English. 

• The team should establish a dialogue with other employees, to 
get their input about what the program should be achieving, help 
them have realistic expectations for the program, and discuss 
what is being - or isn't being - achieved. Co-workers might, for 
example, be invited into class, to see what is going on. 

• At the same time, we need to recognize that Joy is the expert in 
ciuTicultmi design. She will ultimately be responsible for 
designing activities to meet appropriate expectations. 

Outside farilitator's interpretations and 
recommendations : 

See same section under "Question #1" above. 

Question #3: What in fact is being achieved by the 
program? Are Datatec employees now better able to 
communicate in English? 

What our sources said : 

Learners speak in general terms of liking the program, but 
don't articulate specific examples of how they or the company are 
benefitting. 

Some co-workers seem to feel that only the more-motivated 
participant are getting anything out of the program. These co- 
workers seem resentful that training resources are going to the 
less-motivated participants who aren't properly using them. 

Other co-workers, however, feel that learners have come a long 
way in one year. As an example, one new co-worker gets training 
from learners; even if the Spanish-speakers are shy about usmg 
English with her, they give it a try anyway. This co-worker f«els 
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that learners "are doing very well. Some couldn't even say a word 
in English before. Now they're using sentences and having 
conversations." 

Another co-worker said that she sees a big difference in 
learners' use of English on the job. ("They're using it more and 
more each day.") Learners are now writing information on 
worksheets provided by supervisor, tmderstandmg blueprints 
better, using copy machines, and trying other machines. Some 
learners are now also relating better to their children's 
schoolwork. 

This same co-worker noted that now learners are correcting 
her (in a friendly way) when she uses slang. This and the fact 
that learners now talk with her about their kids and other 
personal matters have led to improved communication and morale 
between the learners and this co-worker. 

Team's responses to t he above information : 

• Those involved in the program - direcdy (e.g., learners) and 
indirectly (e.g., co-workers) often are not clear about what is 
being achieved. Sometimes the negative observations of a few co- 
workers push others to ignore the positive results of the program. 

• To help learners clarify what they are achievmg ~ and might 
achieve ~ they might practice articulating what they think the 
purpose of an activity is and then assessing what in fact they've 
achieved. The program might also develop some kind of "tests" 
which enable learners to see what they are in fact achieving and 
where they need more work. 

• Similarly, new activities might be instituted to enable co- 
workers to tmderstand the program ~ and perhaps contribute to 
it rather than just criticize it. For example, co-workers might sit 
down witli learners at limch and help them speak English. This 
could be part of a "language mentoring system" in which learners 
ai'e paired with co-workers willing to volunteer to help learners 
develop their English skills. (Patty will seek volunteers to serve 
as "language mentors.-" As a class activity, learners can prepare 
guideLlnes for mentors, showing them how they can help by, for 
example, "correcting" learners in a polite, constructive way.) 

• Sometimes learners are in fact developing their skills, but are 
afraid to risk embarrassing themselves by making mistakes in 
front of co-workers. This suggests that the level of trust among 
employees is not as strong as it should be. 
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Outside facilitator's interpretations and 
recommendations: 

• Communication is a two-way street, requiring respect, listening, 
patience, clarity by all parties. At Datatec, part of the 
commimication problem might be an insensitivity to learners' fear 
of embarrassment ~ or to outright hostility from certain co- 
workers. The team might delicately raise the question with both 
learners and other employees of "How do we ensure trust 
between ESL students and other employees?" 

Question #4 : What factors are contributing to the 
program's success? What factors are inhibiting 
progress? 

What our sources said : 
Positive factors 

Learners cite: 

• A dedicated, helpful teacher 

• Relevant learning activities (e.g., dictation, asking for 
clarification) 

• Supportive co-workers 

Co-workers cite: 

• Commitment (a "desire to excel") and hard work (a "work 
ethic") by motivated learners, willingness to try even if they are 
"shy." 

• Focus on the basic skills which learners need 

• A good teacher who is continually introducing new learning 
activities (e.g., having learners write out "catches" for the 
company newsletter in which they "catch" (recognize) co- 
workers for doing a good job; or practicing explaining various job 
tasks like cleaning one's eyes in case of an accident, use of 
flammable materials or new machines) to make the program 
relevant. Teacher is "on target" in terms of identifying 
important tasks to focus on and encouraging learners to use 
English with co-workers. 

• A work environment and job-rotation policy which encourage 
all employees to try new things, to get outside traditional 
"narrow" roles of doing only one thing all day long. 
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• Those co-workers who are sensitive to the special needs of 
limited-EngUsh-proficient people. For some coworkers, that 
sensitivity comes from having worked or lived with non-English- 
speakers before, or from perhaps previously Uved as a minority 
or foreigner in another culture. 

• Co-workers who take the time to ask learners to explain what 
they're learning in class and to ask them about their kids and 
other personal interests. 

Tpam members cite: . ^ ^ c ^ 

• When participants get opportunities to try new jobs, they reel 
more motivated to learn. 



Negative factors 

Learners cite: 

• Not enough time for class 

• Other responsibilities (commuting, family, other jobs) which 
make it difficult to study outside class 



Co-workers cite: . 
• Some participants don't try, don't take program seriously (e.g., 



don't study, copy homework, use class as "a coffee break.") ^ 

• Co-workers don't know what program's goals are or what's 
being achieved. 

• After class, participants return to job, continue to speak 
Spanish, and don't understand English-speaking co-workers. 

• Not enough testing or homework is done. Standards are too 
weak. 

• Funds are going to support classes for unmotivated workers. 
Resources should instead go to training and otiier benefits for 
motivated ones. 

• Sometimes when group leaders leave floor to go to class, others 
who rely on that leader have trouble. 

T^ ^TTi members cite: 

• Participants' over-reliance on Spanish dimishes team spirit. 

• Some learners are content with ciurent position, don't see class 
as means to promotion, are thus less motivated. 

• Some English-speaking co-workers have low expectations of 
Spanish speakers, don't expect tiiem to use EngUsh or try for 
promotions. 

• Some Spanish-speakers have low expectations of tiiemselves, 
don't think of themselves as capable of speaking English. 
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• Some participants have more formal education than others. 
Others then have lower expectations of less-educated 
participants. 

• Participants don't get raises, even when tiiey do a good job and 
train new co-workers. Opportunities for promotion are limited. 
Recognition in newsletter is not enough to motivate learners. 

• Participants are told not to complain about problems at work 
on grotmds that they are being given an English class. 

• Too much emphasis on evaluating employees can lead to 
conflict, diminish team spirit 

• Americans tend to not understand the difficulties faced by 
someone with limited English skills. 

• However, some co-workers do want to try to comunicate with 
Spanish-speakers, but ~ if they can't imderstand what the 
Spanish-speakers are saying ~ they find it awkward to have to 
ask the Spanish-speakers to repeat the same thing several times. 

Team's responses to the above information : 

• There was general agreement within the team that "low 
expectations," "cultural differences," and "lack of seriousness" are 
potential obstacles to meeting program goals: 

Low expectations : 

Several team members agreed that some learners have 
low expectations of themselves. This is a barrier to learning. 
These "low expectations of self' are another way of saying "low 
self-esteem" or "low self-confidence." 

Cultural differences: 

It appears that we need to focus more on cultural issues 
which block a sense of being part of the Datatec family. 
Immigrant employees seem to have a sense of separation from 
other employees. (One example: Few immigrant employees 
participate in company's overnight teamwork sessions at Frost 
VaUey.) 

Maybe one reason for this is that traditional Latin 
American culture doesn't encourage women to step outside 
traditional roles. (For example, few husbands would allow 
their wives to go someplace overnight with co-workers.) 

The structure of the Frost Valley event doesn't allow full 
participation by all employees. A possible action: Ask the 
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learners to suggest actions which might enable/encourage 
learners to participate more actively in company activities. 

If this could be done, it would get at the trust issue 
referred to earlier. 

Lack of seriousness : 

If in fact some learners don't take the program seriously 
enough, then this needs to be dealt with. Is it true? If so, 
why? And what can be done about it? 

If learners don't try, they might be asked to leave the 
program. 

• One team member questioned the comment suggesting that 
employees who do a good job aren't given proper 
incentives/rewards. 

• Another questioned why someone would raise the issue of how 
much education learners have. That is not relevant to this 
discussion, she felt. 

Outside facilitator's interpretations and 
re commendations : 

• While some employees might have "low expectations of 
themselves," this low self -confidence might be reinforced by co- 
workers or other factors which convey the message that the 
learner isn't really expected to change or improve very much. 

• While some learners might not take the program very seriously, 
it could be that some co-workers don't take the program seriously 
either. 

• The team might deal with these problems by stressing the 
importance not only of the learners in the program but of co- 
workers as valuable resources. The team might also take the lead 
in promoting the value of education. This might be done through 
buttons ("I'm improving my skills." "Speak to me in English."), 
posters, articles in the newsletter (including articles written by 
learners or interviews with learners, evaluation team members, 
and other supporters of the program), etc. 

• The company also needs to recognize that cultural differences 
might be the source of many problems currently blamed on lack 
of English skills. These cultural differences might be brought out 
in the open in classroom dicussions and through projects in which 
learners interview co-workers in ways which would make cultural 
differences more explicit. 
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O^ IPstinn #5: Should the program continue? 



What our sourres said: 

Wj^f^t learners said: 

"Yes," program should continue. 

What co-wo rkers said: . t. . i 

Three co-workers said that program should contmue, tm only 
if stricter standards are instituted to ensure that (1) only those 
capable of learning are admitted, and (2) participants are 
committed and do the work necessary to succeed. These 
standards could be enforced through "testing" and a stricter 
homework poUcy. If those changes aren't instituted, training 
resources should be re-targeted to other employees who show 
proper motivation. 

Three other co-workers said without reservation that the 
program should continue. 

Team's rpsponses to the above informat ion: 

• Assuming that the program continues, steps need to be taken to 
see that the above concerns are dealt with. These include: 

- Employees shotdd be warned to take the program seriously. 

- People have to find the time to practice or they won't 
progress. (The class should discuss how important it is to 
practice.) 

- The program needs to "put our foot down ' and be strict 
about homework. 

- All stakeholders need to keep in mind that aU learners have 
the capability to learn. Motivation, instruction, and materials 
are factors which determine program success. 

• In addition to tiying to respond to problems identified in tiie 
data, we need to recognize positive factors and build on tiiem. For 
example, the team now sees more motivation among learners in 
recent montiis. Over-emphasizing tiie negative comments of some 
of our sources could be discouraging to tiiose tiying to make tiie 
program succeed. 

Outside facilitator's intprpretations and 
l-p^-^piTTipndations: 
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See this section under "Question #6" below. 



Question #6: If the program continues, what steps 
need to be taken — and by whom — to help achieve 
the program's goals? 

What our sources said : 

What learners said - 

• Provide more time for classes. 

• Find out from learners what activities they like (e.g., dictation) 
and eliminate less-interesting activities. 

• Recognize that learners lead busy lives outside Datatec and 
therefore have limited time for extra study. 

• Learners are willing to continue classes on their break times, if 
that is what is necessary. 

• Co-workers should continue to pay attention to learners, explain 
things clearly, and help them with English. 

What co-workers said: 

• Have stricter standards (e.g., use tests) to determine who gets 
into and stays in the class. Be sure that all participants are 
"capable of learning," and weed out those who don't try (e.g., via a 
"pass/fail" policy). 

• The State should be more careful about how its funds are spent 
and demand that programs meet certain standards. (Note: Larry 
Tourjee points out that Datatec is the only funder for this 
program. It receives no government money of any kind.) 

• Provide training to other workers, too. 

• See that participants do homework, don't copy from each other. 

• Keep co-workers informed about program goals and activities. 

• Current participants might identify others who should 
participate. 

• Participants need to try to use the English being taught in the 
classes. Perhaps they need to be paired with non-English- 
speaking co-workers. 

• Have classes focus on dialogues, interviews, and other activities 
which teach the skills the participants actually need. 

• Encourage learners to use ESL tapes at home and participate in 
adult education classes, to reinforce what is being learned in the 
class. 
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• English-speaking co-workers should help learners, not make 
them feel imcomfortable by being impatient. One co-worker said: 
"If you are mean to a child, she'll avoid you. If you're nice, she'll 
come to you." Another said: "I used to laugh when one of them 
made a mistake in English. Now I won't laugh, just smile." 

• Co-workers can encourage Spanish-speakers to "speak EngUsh!" 
in a fun - not intimidating - way. For example, tell them with a 
smile "I want to understand your gossip!" 

• American co-workers should recognize that limited-English- 
proficient employees can progress tremendously if given the 
chance. The learners are hardworkmg, eager to learn, and want 
the chance to learn. 

Team*s re i^ pnnses to the above information: 

• The program needs to encourage active participation by all 
learners. Some possible ways of doing so: 

~ Institute a pass/fail system. 

~ Get co-workers more involved in the program, so they will 
imderstand it and support learners. Show co-workers how to 
give constructive feedback to learners. 

~ Consider re-arranging work assignments, to encourage more 
commimication between English-speakers and Spanish- 
speakers. 

~ Develop interesting projects for learners to do outside class. 
This will enable them to get the practice they need and to see 
the relevance of what they are doing in class. As a class 
activity, learners can keep track of the homework they are 
doing as an indicator of progress. 

- Recognize learners and the instructor for the good work they 
are doing. Find ways to communicate information about 
program goals, activities, needs, and accomplishments to other 
employees. 

~ Deal with the negative comments raised by some co- 
workers, so that they don't create bad feelings. 
~ Consider allowing more time for class. 1.5 hours per week is 
a small amount of time. 

• Other stakeholders need to re-examine the role of the ESL 
program and attitudes toward cultural difference. The program 
should be seen not just as an isolated effort to "teach English" to a 
few workers. Rather, stakeholders should seeit as part of a larger 
effort to facilitate communication and active involvement among 
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all employees. This communication is being blocked not only by a 
lack of English skills among some workers but by an inability by 
many employees to xmderstand and deal with cxiltural differences. 

For example, some English-speaking employees don't 
understand why - after a year or so of classes - some employees 
still speak Spanish inthe halls. This might not be a sign that 
Spanish-speakers can't use English or that the ESL program is a 
failure. Rather, it might simply be an efficient way of 
commtmicating for those particular employees. 

Stakeholders need to clarify their expectations for the ESL 
program, so the program is not being expected to change deep- 
seated attitudes and communication patterns beyond the control 
of the program. 

Stakeholders should consider that, just as the company has 
been transforming itself technologically, it might have to 
transform how its "culture" handles tiie diversity of its workforce, 
as well. 

• Other stakeholders need to make efforts to commimicate openly 
with Spanish-speakers, as well as vice versa. This might be done 
through a proposed "mentoring" program which pairs English- 
speakers with Spanish-speakers as "buddies." To make a 
mentoring program work, however, mentors need guidance in 
what they should do. 

• Other stakeholders should also be aware that, during the past 
few months, the program instructor has made several significant 
changes in curriculum, assessment, and commimications with 
other stakeholders. These changes were made in part due to 
feedback received during this evaluation. These changes include: 

Curriculum : Extra efforts are being made to focus instruction 
on job-related uses of Egnlish identified as relevant by learners 
and other stakeholders. 

Assessment : The instructor has introduced a "portfolio" 
assessment system in which learners ~ with help from the 
teacher ~ keep track of their progress toward specific learning 
goals. The instructor, in turn, shares summaries of that 
progress with management on a regular basis. 
Communications witii other stakeholders : The instructor has 
been giving regular progress reports to management and 
soliciting input from them. She is also reaching out to other co- 
workers, inviting them to get involved in the program by 
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serving as language mentors. The planning team is also 
considering inviting other co-workers to be part of the team. 
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Outside farilitator's interoretations and 
recomme ndations: 

The preceding comments from learners, co-workers, and 
planning team members ai"e a mixture of praise, creative ideas, 
constructive criticisms, and complaints. The team needs to sift 
through these ideas, acknowledge legitimate concerns, and take 
needed actions. 

In summary, the team might take the following positive 
actions: 

Build stronger partnership and communication among 
all stakeholders. 

The team should develop ways to regularly get input, 
feedback, and support from management, co-workers, and 
learners. By so doing, these various stakeholders can clarify 
what they expect the program to achieve, whether the program 
is on target, and how to improve it 

Possible ways of improving commimication include: 

• Fxpand the number of stakeholders on the planning team . 
The team might include 1-2 supervisors or other co- 
workers, and another 1-2 managers in charge of quality 
and/or personnel issues. These people need not attend 
every team meeting, but be available to clarify questions 
which might arise in team meetings. For example, when 
setting program goals, the team might want a clearer idea of 
how education fits into the company's "quality" program. 
Supervisors might provide specific examples of 
commimication problems which the program might help 
with. Or learners might have questions about about benefits 
available to them. 

• Host periodic "open houses " in which co-workers are 
invited to sit in on class, hear reports from learners, respond 
to questions from learners, etc. 

• E^rnHi ^^ a "language me ntoring" program in which 
interested co-workers are matched with one or more 



learners to provide feedback, explain procedures or 
vocabulary, etc. 

• publish new^lpttfir article s about the program. The 
learners and instructor might write their own articles for 
the company newsletter. Or the newsletter editor might 
profile individual learners or describe class projects. This 
would clarify to other employees what is happening in the 
program. Alternatively, the class might compUe their own 
collection of articles, "action plans," autobiographies, etc. for 
dissemination witiiin the plant 

• Dpal px plicidy ^r^ ^b<> "trust" issue. As a class exerase, 
raise the question of "trust." That is, why is trust important 
for all employees? What inhibits trust among co-workers? 
How can trust be fostered, so tiiat leaxners feel confident • 
about dying to use English? 

• ppal px plicitW ^A;^th poss iblp ro-worker resentment. 
Feedback from some co-workers suggests that some might 
in some way resent the ESL program and/or some learners. 
The team might open up a dialogue with co-workers around 
tiiis issue, to deal witii it directiy. For example, in a meeting 
with co-workers, tiie team might ask whether some co- 
workers feel resentful. If so, why? What might be done 
about it ~ and by whom? 

• pp;^1 ex plicitly with cultur al differences which might 
inhibit mmmnniration an d teamwork. The team should 
consider to what degree differences between learners' 
culture and that of other co-workers might block 
communication and cooperation within the plant. If such 
differences are identified by tiie team, tiiey might be dealt 
with in classroom activities, in discussions witii co-workers, 
by revising certain plant activities (e.g.. Frost VaUey, how 
learners are treated in teams, etc.) Perhaps learners can 
delegate one or more of tiieir fellow learners to serve as 
"cultiiral ambassadors" to tiie rest of tiie workforce. The 
"language mentors" mentioned above might play a similar 
role of facilitating communication between learners and co- 
workers. 

Clarify expectations and monitor progress. 

The above activities should help all stakeholders clarify 
expectations for the project. Otiier ways of doing so might 
include: 



19 145 



• Establish a portfolio assessment system in which learners 
work with the instructor to continually clarify goals and, in 
various ways, docimient progress toward those goals. 

• Demonstrate results in a dear, quantified, graphic way . 
Where possible, document learner progress toward goals in 
charts or other graphic forms familiar to Datatec co-workers. 

• Analyze what role com munication p lays in the plant. The 
team might conduct an organizational needs analysis to 
clarify what activities really requii*e reading, writing, 
listening, and speaking skills. This analysis should also 
analyze written materials used in the plant, to clarify how 
they might be written more clearly to make them more 
accessible to learners and other employees. This would help 
the team clarify to what extent communication is blocked by 
learners' English skills or by other factors like poorly-nm 
meetings, poorly-written documents, etc. 

• With input from learners and other stakeholders, establish 
a set of "ground rules" or guidelines which all learners need 
to abide by. These might include doing homework, regular 
attendance, showing progress, etc. Learners should then 
formally agree to abide by those rules via a learning 
contract. 

• Establish a fair set of guidelines for clear communication 
and teamwork . With input from learners, the team might 
prepare a set of guidelines for using English in the plant 
Guidelines might emphasize the links between clear 
communication, teamwork, and quality. For example, they 
might go something like this: 

"Datatec emphasizes quality work in a team environment. 
Clear communication is key to making this work. All 
employees should use English to communicate with co- 
workers in a clear way, to identify problems, and jointly 
solve them. Communication and teamwork are two-way 
streets, requiring respect and extra effort to speak and 
write clearly and to listen carefully ..." 

Revise curriculum content. 

Revise the curriculum to focus more directiy on 
competencies and activities identified as important in the 
above goal-clarification activities. This might include asking 
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learners and co-workers to identify issues they are most 
concerned about (in the plant or outside). Learners can then 
draw up action plans analyzing the problems and 
recommending possible solutions. 

Consider altering schedule of learning activities. 

The team should consider whether it is necessary, 
desirable, and feasible to expand or otherwise change the 
scheduling of classes. Note that many learners asked for more 
class time. Can this be done in ways that are not disruptive to 
production schedules? 

The team should also re-think how learners might be 
given opportunities to practice their skills outside the 
classroom. They might, for example, be given assignments to 
do on the job or after work. They might also be referred to 
other eductional programs, TV shows, reading materials, etc. 
which they could use outside work to meet their learning 
objectives. 
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THE HAARTZ CORPORATION 



Acton, Massachusetts 



I. PROGRAM BACKGROUND 
Program history 

The Haartz Corporation is a small {250 employees) 
manufactiirer of custom, vinyl-coated, commercial upholstery fabrics. 
These fabrics are purchased by manufacturers of cars and boats and 
installed in vehicles that are sold worldwide. Like most small 
businesses, Haartz has been trying to keep production costs down 
while it improves product quality. The reliable quality of its 
products and its ability to respond quickly to custom orders keep 
this energetic company at the center of its industry. 

A few years ago, the President of the company and other 
managers began to see the need to provide basic education support 
to some workers. The changes in production technology and 
organization of work that were contributing to the company's success 
in the marketplace were also highlighting another reality: a 
significant number of workers could not master the new production 
technologies and work structures because their basic English 
language and math skills needed improvement. The new work 
structures which highlighted the need for workers to improve their 
skills include: working in teams, emphasizing creative problem 
solving at each phase of work, creating flexible job roles, 
implementing statistical process control measures, and becoming 
generally oriented to quality management. 

The Haartz Corporation is an unusual company. It is small, 
family-owned, profitable, and managed by a young man in his late 
thirties whose father and uncles ran the business before him. This 
young President and other managers like him have succeeded in 
making team management, collective problem solving, and flexibility 
more than slogans. Employees are motivated to learn and to change 
because they are invested in the success of the business of which 
they feel themselves to be a valued part. While there are 
acknowledged problems with the quality of work some employees 
can produce because of their limited skills, the general attitude 
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toward employees is not condescending but supportive. The cultural 
message in this organization is that education will help employees in 
their own lives and it will help the business -- in whose profits the 
employees share. 

Program goals 

The team enjoyed the luxury of a six-month planning period 
before conducting classes. During this time the team worked hard to 
understand exactiy what their basic skills program should achieve. 
The Program Coordinator was familiar with the SCANS Report and 
other reports like it which suggest that American business needs to 
reorganize itself by, in part, training workers to become more 
flexible and multi-skilled. As a result, the team began to align the 
goals and practices of the basic skills program with the mission 
statement of the company ~ and actually rewrote that company 
mission statement in response to the new thinking which the basic 
skills program provoked. 

When the evaluation facilitator joined the team, she 
encouraged team members to further review the program's goals 
from the perspectives of the workers, the company, and the program 
itself, and to pay attention to the similarities and differences among 
those goals. The charts on pages 9-13 document several iterations of 
program goals and indicators which the team produced. 

Description of Program 

With assistance from a grant through the Massachusetts 
Workplace Education Initiative and the National Workplace Literacy 
Program, the Haartz Corporation offered language arts and math 
instruction to about 50 employees over an 18-month period from 
July 1992 to December 1993. After the initial planning period of six 
months, classes were offered in twelve-week cycles. Two math and 
two language arts classes ~ all an hour and a half in duration ~ 
were offered in each cycle and sch-^duled morning, midday, late 
afternoon, and night so that employees from all shifts could attend. 
Employees received full release time from work to attend classes. 
Employees were enrolled in only one course at a time. 

The Haartz Corporation's educational partner in basic skills 
education is the Acton Public Schools/Acton-Boxborough Regional 



School District. Since January 1994 the company has been 
supporting the basic skills program with its own fund. 

In addition to providing basic skills education, the Haartz 
Corporation also provided GED instruction to twelve employees 
during the second half of the grant period. These services were not 
directly tied to the grant but grew out of an increasing recognition of 
a need for them as the math and language arts classes became more 
popular. The company soon made a special arrangement with Mt. 
Wachusett Community College and the Commonwealth of 
Massachusetts GkD Program Office to become an official GED test site. 
This made it possible for the twelve candidates for the GED to take 
the test in their own company. This was a strong symbol of the 
workplace becoming a learning place. 



11. BACKGROUND ON THE TEAM 

History of the team 

As required by the Massachusetts Workplace Education 
Initiative (MWEI), the Program Coordinator convened a Planning and 
Evaluation Team composed of representative stakeholders in the 
summer of 1992. The MWEI required all its 1992-93 funded 
programs to convene such teams and to follow a planning and 
evaluation process virtually identical to the process being piloted in 
this National Institute for Literacy (NIFL) project. 

The reason for this overlap is that the evaluation model being 
piloted under this NIFL project grew out of work done in 
Massachusetts. The MWEI pursued its work in team-based 
evaluation on a parallel path to the work being done for NIFL. 
Rather than present a conflict, this convergence of interests created 
an opportunity for the evaluation facilitator to conduct her work in a 
program already required to conduct evaluation activities with a 
team. The facilitator was able to provide guidance in evaluation to 
the Haartz team (in addition to the support it received through its 
participation in the MWEI) for about nine months, and to document 
the work for the purposes of this case study. 
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Stakeholders represented on the team 

The Haartz Planning and Evaluation Team included: 

• Eric Haartz, President, the Haartz Corporation 

• Doug Seibert, Plant Manager 

• Fred Carley, Quality Control Specialist, On-site Coordinator 

• Mike Krupski, Supervisor 

• Nelson Purdue, Supervisor 

• George Pickles, Employee 

• Rodney Goguen, Employee 

• Russel Tysinger, Employee who replaced George Pickles and 
Rodney Goguen 

• Arthur Goodall, Program Coordinator, the Acton Public 
Schools/Acton-Boxborough Regional School District. 

• Jane Goodall, English Instructor, the Acton Public Schools/Acton- 
Boxborough Regional School District. 

• Ken Russell, Math Instructor and Haartz Extern 

Note that not all members of the team attended each meeting. 
The "core team" (especially toward the end of the evaluation period) 
was composed of Fred Carley, Art Goodall, Jane Goodall, Ken Russell, 
and Russel Tysinger. Eric Haartz attended more than half of the 
meetings. 



III. THE PROCESS WHICH THE TEAM WENT THROUGH 

Summary: Between December 15, 1992 and August 30, 1993 the 
facilitator met with the Haartz team approximately twice per month 
for a total of 16 meetings. Each meeting ran for about two hours. 
Team members often discussed other business before getting down 
to the work of the evaluation. This helped the evaluator to become 
"situated" in the program and to get to know key team members in 
their program-related roles. In addition, the facilitator 
conmiunicated frequently with team members by phone ~ 
especially the Program Coordinator and the On-site Coordinator- and 
met and talked with team members at several conferences on 
workplace education. 

After clarifying its program goals, the team chose to focus its 
evaluation on whether the basic skills program was improving the 
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communication skills of workers enrolled in the program. It did so 
after long and interesting discussions about the relative merits of 
focusing an evaluation on formative versus summative questions. 

After deciding that it would focus on the improved 
communication skills of enrolled employees, the team spent several 
meetings identifying the varied indicators of improved 
communication skills at the Haartz Corporation. It planned to 
evaluate improved communication skills through several measures: 

• a pre-post survey administered to students 

• a student focus group 

• a supervisor focus group 

• a program log 

• informal feedback from the Haartz community 

Other goals were also evaluated through these and other 
measures, as indicated in the Goals Matrix on pages 27-28. 



Phase 1 : Preparing the team 
Initial preparations 

As mentioned, the Haartz Corporation workplace education 
program was part of the Massachusetts Workplace Education 
Initiative (MWEI). MWEI staff prepared its member programs for 
team-based evaluation through a three-hour orientation. When 
Laura Sperazi joined the Haartz team as evaluation facilitator, 
members were ready to work, although they were somewhat wary 
about the amount of time that a team-based evaluation might take. 
The Haartz Corporation had also made several trainings in teamwork 
available to supervisors early in the grant period. Supervisors on the 
team had participated in this training. Overall, the Haartz Planning 
and Evaluation Team was well-prepared to work as a team. 

Organizing the evaluation team 

The evaluation facilitator did not have to organize the team. It 
was in place when she began her work with the company. It had 
been convened according to MWEI guidelines which required the 
representation of all stakeholders. However, the evaluation 
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facilitator encouraged the participation of additional workers who 
were enrolled in the prograni. 



Phase II : Planning the evaluation 

It took roughly three months for the team to plan its 
evaluation. During that time it reviewed its goals and discussed the 
relative merits of focusing on formative versus summative questions. 

Haartz is a quality-oriented company. It is committed to 
producing the best-quality products possible. The company follows 
the general quality directive (promoted by Edward Deming) that, as 
long as manufacturing processes are operating within quality range, 
there is no great need to focus excessively on the quality of your 
outcomes. In other words, quality outcomes are inherent in quality 
processes. Quality processes guarantee quality products. Because 
this orientation to creating a quality product works for 
manufacturing at Haartz, it made sense that the team would examine 
its education program ~ its educational product, if you will ~ in the 
same way. 

The question thus facing the team was: Does the team have to 
concern itself with measuring outcomes? Or, if members were clear 
about the outcomes they wanted to achieve, could they focus simply 
on their "processes," on making a better and better program? This 
leaning toward an exclusively process-oriented evaluation was 
reinforced by the amount of anecdotal summative information about 
students' achievements that circulated throughout the organization. 
There was a sense that everyone, including supervisors, knew 
enough about what the program was achieving. They wanted to 
maximize the achievements by concentrating on making a better and 
better program. 

The team came very close to dismissing formal summative 
evaluation activities. In the end, however, some more- traditional 
pressure to demonstrate what the program was accomplishing won 
out. This happened in part because the company would be required 
to demonstrate more frequently to its customers exactly what its 
quality management and other training programs were achieving. 

The team received additional evaluation support from three 
sources: 
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First, selected team members met with members of two other 
evaluation teams (from Bull Worldwide Information Systems' 
Brighton Plant and the Norton Company in Worcester, Massachusetts) 
to share ideas about how to focus an evaluation of a workplace 
education program. 

Second, anticipating the need to present the results of their 
evaluation in statistical form, selected members participated in a 
day-long training on the use of the Statistical Package for the Social 
Sciences (SPSS), a database program. 

Third, the team received technical assistance on survey design 
from Donald Cichon of Donald Cichon Associates. The team wanted 
the survey to present simple and easy-to-read graphic 
representation of items. The team settled on a format that is more 
graphic than most surveys. 

Clarifying program goals and the information to look 
for 

Over the course of approximately four meetings, the team 
discussed and refined its overall goals, chose one goal to evaluate 
(unproved communication skills), and specified about twenty 
indicators of that goal. The facilitator led the team activities in these 
meetings by (1) suggesting that goals be articulated from the 
perspectives of different stakeholders; (2) clarifying the differences 
between goals and indicators; and (3) providing the team with ample 
opportunity to brainstorm and discuss possible goals and indicators. 

As mentioned, the team had been working on its goals for some 
time; their work with the facilitator built on that foundation. 

It is also important to clarify that the team did not hesitate to 
hold students in the math class acountable to the goal of improved 
conmiunication skills, in the same way that students that in the 
language arts class would be. Although the content of one course 
was math, the team fully expected ~ and the instructor was 
confident ~ tiiat the math students should be evaluated on their 
communication skills. Focus group results confirmed the 
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communication skills which students learned in the matli class. (See 
below, pages 17-22.) 

The following charts present the goals for the Haartz program 
and the indicators of improved communication skills. Although 
somewhat lengthy taken as a whole, the charts render an interesting 
picture of the sometimes circuitous process which a team must go 
through in order to identify meaningful and representative goals and 
accurate indicators. 

(Please note that in Chart #2 and Chart #3 "indicators" 
sometimes are actual indicators and sometimes are possible 
"measures" of the stated goal. Chart #4 lists only indicators for the 
goal "improved communication skills.") 
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Chart #2: Haartz Corporation Basic Skills Program Goals and 
Indicators by Program/Empioyee/Organization: 
First Iteration 



Program Goals 

L Viewing student as "Customer," we 
want to offer them a high quality 
product. 



II. Use "SCANS" competencies and found- 
ation skills to create curriculum tailored 
to needs. 

III. Allow for personal growth and devel- 
opment of employees by enhancing abil- 
ity to perform and build confidence. 

IV« Make life-long learners of students. 



V, Provide feedback to DOE Office of ABE 
in terms of start-up procedures and 
curriculum building. 

VL Build program based on student input 
(classroom feedback and talk on shop 
floor). 

Use experience gained from Phase I of 
program to make this program effective 
and efficient. 



Indicators of Goal Attainment 

Student satisfaction with course and 

self-performance. 

"Customer" desire to continue using 

product. 



Demonstrated ability and desire to 
impro\ e job status. 



Student desire to continue education, in- 
house and/or outside. 

Complete instructor notes and lescon 
plans along with comments and recom- 
mendations. 

Student satisfaction that course is 
tailored to needs and capability. 
Student sense of accomplishment. 



Student/Employee Goals 

L Improve self-confidence of students to 
make better employees. 

II. Improve communications. 

IIL Additional goals as communicated by 
students to instructors. 

IV. Student satisfaction. 



Indicators of Goal Attainment 

Mid-course and post-course critiques. 
Supervisor evaluation. 

More active involvement in discussion, 
and presenting ideas. 
Supervisor evaluation. 

Mid-course and post-course critique. 
Problems brought to class as being of 
interest to students. 

Mid-course and post-course critiques. 
Improved performance. 
Students spreading word of how helpful 
course is. 
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Chart #2: Haartz Corporation Basic Skills Program Goals and 
Indicators by Program/Worker/Organization: 
First Iteration (cont'd) 



Or ganization Goals 



I. Increase skill level of students. 



IL Improve communications, includ- 
ing with supervisors. 



III. Improve productivity. 

IV. Improve self-confidence. 



V. Improve problem-solving skills 
with students to encourage their 
coming forward with solutions. 

VI. Vertical/Horizontal team-building 
and problem solving. 

VII. Use program as stepping stone to 
establish GED attainment program. 



VIII. Establish within the culture of 
Haartz Corp. a dynamically evolving 
philosophy of Hfe-long learning. 



Indicators of Goal Attainment 

Direct measure of performance in 
subject being taught. 
Subjective measure of daily perform- 
ance by supervisor. 
Performance evaluation interview. 
End-of-course critique. 

Direct measure of performance in 
class. 

Subjective measure of daih- perform- 
ance by supervisor. 
Performance evaluation interviews 
End-of-course critique. 

Supervisor evaluation. 

Student self-evaluation. 
Supervisor evaluation. 
End-of-course critique. 

Student self-evaluation. 
End-of-course critique. 
Supervisor evaluation. 

Supervisor evaluation. 



GED attainment statistics. 

Percent of employees with H/S diploma 

or GED. 

Employee response to continuing 
program with more diversified 
subjects. 

Employee interest in continuing 
education outside workplace. 
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Chart #3: Haartz Corporation Workplace Education Goals and Indicators: 
Final Iteration 



Program Goals 

• Viewing student as "customer," 
we want to offer them a high 
product 



Organization Goals 

• To improve problem-solving and 
communication skills with students 
to encourage their coming forward 
with solution 



To support and enhance Vertical/ 
Horizontal team concept 



To establish within the culture of 
Haartz Corporation a dynamically 
evolving philosophy of life-long 
learning and to create a bridge to 
other programs such as GED 



Indicators of Goal Attainment 

• Student satisfaction with course 
and self-performance 

♦ "Customer" desire to continue 
using product 



Indicators of Goal Attainment 

• Student self-evaluation 

• End of course critique 

• Supervisor evaluation 

• I ncrease i n constructive 
suggestions 

• Supervisor evaluation 

• Team performance improvement 

• Enhancement of team concept 

• Employee response to continuing 
program with more diversified 
subjects 

• Employee interest in continuing 
education outside workplace 
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Chart #4: Indicators of Improved Communication 

1 . Speaks up in meetings. 

2. Says what is really on his/her mind in meetings. 

3. Uses technical words when s/he needs to. 

4. Makes suggestions about how to improve how work is done 

5. Asks his/her supervisors questions when s/he needs to. 

6. Asks the people s/he works with questions when s/he needs to. 

7. Talks to his/her supervisors with confidence. 

8. Lets supervisors know that s/he has important skills which s/he is ready to use. 

9. His/her supervisors understand him/her when s/he talks to them. 

10. The people s/he works with understand him/her when s/he talks to them. 

1 1 . Talks with his/her supcr\asors about ideas that improve quality. 

12. Talks with people s/he works with about ideas that improve quality, such as better 
instructions for Standard Operating Procedures. 

13. His/her supervisors let him/her know s/he has good ideas. 

14. The people he/she works with let him/her know he/she has good ideas. 

15. Reads job instructions. 

16. Reads safety procedures. 

17. Reads bulletin boards. 

18. Works well with other people on the floor. 

19. Offers help to his/her supervisors. 

20. Offers help to people s/he works with. 

2 1 . Teaches new workers about how the workplace runs. 

22. Uses the information on the floor (i.e. computer printouts, memos, etc.) to 
improve how s/he does his/her work. 

Note: See Goals Matrix on pages 27-28 for summary of goals and evaluation results. 
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Phase III: Gathering, organizing, analyzing, and 
reporting the information 

Gathering and organizing the data 

The evaluation facilitator worked with the team to gather and 
organize its information, using the following activities: , 

• Pre-Post Student Survey. With input from the team, the 
evaluation facilitator developed a draft pre/post survey to collect 
information on improved communication skills. The team revised 
the instrument. Then the evaluator drafted a new version. This 
process was repeated four times before the survey was put into 
final form. 

The survey has twenty-six items; four are demographic and the 
rest are indicators of improved communication. Each item offers 
the student a choice of one of five answers (i.e., not at all; 
sometimes; a lot; always; does not apply) with corresponding bar- 
graph-type representations. At the end of each item, the student 
is invited to comment in writing on the question asked/answer 
given. The evaluation facilitator also drafted a version of this 
survey for use with students who had not taken the survey as a 
pretest. This version asked students to assess each item twice ~ 
once for before they began the program and again for after the 
program. (See Appendices A and B.) 

The Program Coordinator piloted the survey with two students 
for content validity and reliability. (See Appendix C.) Teachers 
administered the survey in class. The evaluation facilitator 
reviewed and organized the raw data. 

• Focus Group. The evaluation facilitator and the Program 
Coordinator developed questions for the focus group for workers 
who were participants in the program, and then conducted the 
focus group together. The evaluation facilitator wrote up the 
results of the focus group. (See Appendix D.) 

• "Math Midcourse Critique." The math instructor developed 
and administered the "Math Midcourse Critique." This written 
feedback sheet is composed of eight multiple-choice questions 
which assess how well the students feel the course is going for 
them: speed of the course; clarity of instruction; quality of the 
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materials, usefulness of what is being taught, etc. There is space 
for comments at tiie bottom of the survey. 

The math instructor administered the "critique" in class in 
order to gather feedback from students about what they liked and 
did not like in class. It was intended to be an anonymous, quick, 
and easy check on "customer satisfaction" whose results could be 
tallied without having to analyze narrative answers to many 
open-ended questions. (See Appendix E.) 

• The Program Log. The evaluation facilitator provided 
guidelines to the On-site Coordinator about how to keep a program 
log that would provide documentation ~ become a compendium 
~ of the rich, oral, anecdotal information about the program that 
flowed freely throughout the organization. The On-site 
Coordinator was responsible for implementing the program log. 

A program log is a book or folder where ideas and stories about 
a workplace education program can be recorded, preserved, and 
then reviewed. The log is the journal of the program, kept 
collectively by students in the program, workers who are not 
students in the program, teachers, supervisors, managers, 
administrators ~ anyone who cares enough to think about the 
program and have an idea worth putting down on paper. The 
purpose of keeping such a log is to preserve valuable information 
that might otherwise get lost. The log is a continuous 
improvement tool. It is the record of the program's "production." 
After you review the production record, you are better informed 
to make decisions about how to improve operations. 

A team member (usually but not necessarily a teacher) is 
responsible for reviewing the log, taking the log to team meetings, 
and reporting interesting findings to other teaiii members. Every 
few months the "keeper of the log" writes a brief summary of 
what has been recorded in the log, and this report, in turn, 
becomes part of the log. Program staff, team members, and others 
should be able to review a year's activity fairly easily this way, 
once the system gets going. 

• Supervisor Focus Group. The team generally agreed that a 
supervisor focus group would be an important complement to the 
worker focus group. However, there was also concern about 
taking supervisors away from work for an hour, especially when 
some of the supervisors were already freely sharing their 
opinions about the program either in team meetings, in their own 
meetings, or in conversation with each other and other managers. 
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The team did not convene this group during the time the 
facilitator worked with it. 

• Newsletter Articles. The team considered regular articles or 
updates written about the basic skiUs program for the Haartz 
Corporation's monthly newsletter to be part of the evaluation 
documentation process because the articles req\ured students 
with the support of teachers - to think through Jie most 
important issues to communicate about their program. These 
"updates" keep everyone in the company informed about the 
program - who is in it, what is changing, why others should join. 
Even in a company as enlightened as Haartz, this publicity 
legitimates the program and helps to diminish the stigma attached 
to (as a worker in the focus group said) "having to go to school 
now because you're a dummy." Participants in the worker focus 
group considered these newsletter updates very valuable. They 
are written by students in the program as part of a class activity. 

• The Process Evaluation. FoUov^ng a chart which lists 
seventeen components of a quality workplace education program 
and "quality indicators" for each of those components, members of 
the Planning and Evaluation Team attempted to set quality 
standards for each component of the Haartz program. (This chart 
of "quality indicators" was prepared by the Massachusetts 
Workplace Education Initiative and was based, in part, on the 
facilitator's early work with evaluation teams.) The team 
determined whether the component they were evaluating 
maintained a quality standard or needed to be improved. If 
improvement was needed, the team then identified the actions it 
needed to take to make that improvement. The team met with 
the facilitator for two meetings to complete this part of the 
evaluation process. (See Appendix F.) 

Analyzing the data 

Once the above data-gathering activities were implemented, 
the team analyzed the resulting information, as follows: 

• Pre-Post Survey. For all the work that the team put into 
designing the pre-post communications skills survey, in its first 
round of use (in the second cycle of classes), it did not yield valid 
results. This is because students did not evaluate their 
communication skills objectively at the start of the class for fear 
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of "looking bad." Although the math teacher had doubts that the 
pre-test data were accurate -- he suspected that the men 
especially were inflating their self-assessments of their 
communication skills because they did not want to "look bad" -- it 
was not until after the post-test that some of the students (and 
the data) confirmed his suspicions. 

Many students rated their communication skills at the start of 
the program very high. At the end of the program they rated 
their skills more realistically and many overall scores showed a 
loss instead of an anticipated gain. The lower scores can be 
viewed positively as a more-honest assessment of students' 
communication skills rather than as a failure to achieve gain. That 
honest assessment means that something valuable took place both 
in the education process and in the evaluation process. It means 
that ( 1) students felt free to discuss the motivations of their 
earlier asssessments, thereby demonstraing their improved 
communication skills; and (2) the survey instrument registered 
the discrepancy between anticipated and actual results in a way 
that promoted critical self-reflection by all parties. 

Related to survey results is the use of the database "Statistical 
Package for the Social Sciences" (SPSS). As mentioned earlier, 
training in SPSS had been provided to the team. The facilitator 
and team members expected that the math instructor ~ who had 
a lot of experience with computers and statistical analysis in his 
prior career ~ would learn SPSS and subject survey results to 
statistical analysis with relative ease. The math instructor had 
agreed to this mid-way through the evaluation process. However, 
a scarcity of available personal computers in the company, 
lingering questions about whether the small number of surveys 
warranted such analysis, invalid results on the first round of data 
collection, and the additional training the instructor needed on 
new personal computer systems, hampered use of SPSS. 

The teachers and the students themselves knew that a great 
deal of learning had taken place during the second cycle of classes. 
The challenge was to find a format in which students could 
describe tliat change fully. The worker focus group was that 
format. 

• Worker Focus Group. The purpose of the worker focus group 
was to elicit students' ideas and feelings about (1) what they had 
learned from participating in the program, with a special focus on 
communication skills; and (2) how the program might be 
improved. The group elicited very useful information about what 
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students learned; how they learned; what they want to continue to 
learn; and how their learning affected how they do their jobs. 

Focus group participants wholeheartedly enjoyed taking both 
the math and language arts classes. They especially talked about 
how being treated like an adult in the classroom encouraged them 
to become confident about learning. They specifically identified 
the teaching method that uses multiple strategies to solve 
problems as helpful to them. Through the support and ingenuity 
of their instructors, participants have realized that "there's more 
than one way to learn" and have become more familiar and 
comfortable with their own learning styles. 

Focus group participants talked about the benefits of learning 
together. In class, participants become friends, and friends work 
together on the floor more easily and efficiently than people who 
are simply "coworkers." Focus group participants strongly 
suggested that Haartz employees who are not enrolled in the 
workplace education program ~ foremen and supervisors as well 
as workers ~ would benefit from exposure to the team-based 
learning strategies of the classroom. These include; listening well; 
taking people's suggestions seriously; paying attention to alternate 
solutions to problems; not pulling rank if someone has a hard time 
learning something, but being patient and supportive. 

All focus group participants wanted to continue their classes. 
They generally agreed on tiie following actions which the program 
might take to improve itself and to become better integrated into 
the life of the Haartz organization: 

- Continue the workplace education program. 
~ Organize a few open houses and invite foremen and 
supervisors to them. Do not invite foremen and supervisors 
into regular classes whenever they want. This would disrupt 
the classes. 

-- Hold more-frequent focus groups ~ every three months, at 
least, to allow for input on how the program is developing. Get 
all program participants involved in the focus groups ~ not 
just a selected group. 

~ Supervisors should sit in on classes from time to time to get 
an update on what's going on, and to offer their opinions on the 
class. This could be in addition to the open houses. 
~ Supervisors should also have a focus group like the one held 
with program participants. They should have an opportunity 
to say what they think and learn more about the program. 
~ Re-institute the suggestion box. 
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This focus group provided the most detailed and insightful 
information of the evaluation. Further results are detailed here: 

Wore classes eniovable? 

Focus group participants agreed that classes were very 
enjoyable because they were offered on an "adult-to-adult 
basis." Group participants defined adult-to-adult in several 
ways: (1) teachers employ multiple strategies to solve 
problems; (2) teachers use humor or stories to allay 
nervousness or tension participants might feel about not being 
able to learn something quickly; and (3) teachers never "pull 
rank" and make you feel bad for not knowing something. 

Several participants described how ineffective former 
teachers in traditional school settings had been because they 
were unable to approach problems from multiple perspectives 
or take the learning styles of the students into account. 
Participants' comments about why they found classes so 
enjoyable provide insight into why and how adult education 
differs from what we consider traditional classroom teaching ~ 
and why and how it succeeds when traditional methods fail. 
According to participants, traditional teachers are 
unimaginative and don't listen to their students. 

There was general agreement tiiat making a course 
outiine available to students was very important. It seemed to 
give participants a sense of control over their learning. The 
teacher trusted them enough to give them a plan. He didn't 
just spring something new on tiiem each time the class met. A 
sense of having new material imposed on you, when you were 
unready and lacking confidence, seemed to characterize most 
participants' prior school experience. 

nid classes hel p vou at wo rk? If ves. how? 

Participants said that class helped them with specific 
tasks like cutting yardage, preparing formulas, and measuring 
weights. They specifically noted help witii conversion to 
metrics and knowing what numbers mean as opposed to 
performing meaningless functions - on a calculator, for 
example. Learning became a game ~ a literal one where 
participants would bring a problem they had encountered at 
work to class to try to stump the teacher. Eventually everyone 
would try to solve the problem. This fostered a team approach 
to learning and working that the participants appreciated 
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deeply. In fact, this team approach helped participants so 
much that it reportedly created a difference in work-related 
social skills between class participants and others. Class 
participants know how to listen, and ask and answer questions 
better than non-participants. 

According to focus group participants, employees who are 
not enrolled in the class ~ supervisors and foremen as well as 
other workers ~ would benefit from the kind of training they 
received. Focus group participants were very clear that 
learning with others in a supportive environment creates 
relationships which make working with each other much more 
pleasant and efficient. This is the result of feeling that you are 
dealing with a friend ~ someone whose best interest you have 
in mind ~ and not simply with a co-worker, someone who's 
there just to get the work done. 

One participant gave an example of how a supervisor 
criticized something the participant said to him as "nagging" or 
intrusive. The participant answered that it was part of his job 
to make suggestions about how to do things better. The 
participant suggested that if the supervisor had been in the 
class vvith him ~ if the supervisor had had the benefit of 
learning in an environment where offering suggestions is 
welcome ~ he would have heard the suggestion differendy: 
not negatively but positively. This participant said that now 
(since enrolling in class) he is more concerned about quality. 
"My name is on that ticket. And my profit." 

Do vou want others to enroll in class? 

Participants agreed that other workers would benefit 
from enrolling in the workplace education program. They 
talked about what might prevent others from enrolling ~ even 
when others know they need the help and want to enroll. 
Focus group participants said that some people are afraid to 
enroll because it makes pubUc the fact that they need to 
continue their education. They said that some workers make it 
hard for others to acknowledge that they need educational 
support by ridicuUng tb.em - calling the class 'kindergarten." 

Focus group participants said diat there are still some 
foremen who do not understand or support the program, 
making it difficult for workers to choose to go to class instead 
of staying at work. Focus group participants also suggested 
that top-level management needs to "get the word out" about 
the program. Management needs to publicly support 
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enrollment as well as advertise the availability of classes 
through bulletin boards and through foremen's meetings and 
notes. 

Has participation in class improved vour communication ski lls? 

Focus group participants agreed that participation in 
classes has resulted in improved communication skills. They 
specified results like not being afraid to talk; improved 
vocabulary (especially technical words); improved 
understanding of what someone else is saying; and improved 
ability to ask others to clarify what they are saying. 

Related question: Do classes he lp the company? 

Focus group participants agreed that participating in the 
class helps the company. Not only do participants improve 
specific communication skills like the ones described above, but 
participants feel more friendly toward their co-workers. This 
changes the way they perform their work together. 

What would vou like the progra m to offer next? 

Participants were very clear they are ready for the 
following: 

~ higher math 

- more GED instruction 

— college courses. 

Are work-related materials useful? 

Focus group participants agreed that it is good to use 
work-related materials in class, in botii the math and language 
arts classes. The benefits include: learning your own job 
better; learning another job; and having an opportunity to 
practice your own or a new job more than you would be able to 
otherwise. 

Are there anv problems with the classes? 

Focus group participants identified only one problem 
witii the classes: having people at three different levels in one 
class. Participants were very generous with their comments 
about what it means to be in a class where people are at very 
different starting points, but tiiey did say that it is a problem. 
Sometimes having three different levels of students holds 
people back who are ready to move on. Sometimes it makes 
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people feel bad about holding other people back. It means that 
the class is often geared toward the middle. 

Finally, the focus group participants made it perfectly clear 
that their communication skills improved significantly as a result 
of participating in the math class, not just the language arts class. 
This was the result of the math instructor's highly interactive, 
problem-solving approach to teaching. Students had no choice 
but to make their interests and questions known. 

• Math Midcourse Critique. Students liked the math 
midcourse critique. It gave them a chance to tell the math 
instructor anonymously what they thought of the class, and 
helped to build investment in the course as something that 
belonged to them. 

Eleven students completed the critique in the middle of the 
first cycle of classes. The instructor felt that responses were 
honest. For each question, responses tended to cluster in the 
middle range of choices with one or two responses at either 
extreme: (1) students were generally satisfied with the course; 
(2) the course was moving at just about the right pace, but two 
students felt left behind; (3) the instructor explained things 
clearly, but two students said they were not sure they understood 
everything; (4) the material was challenging, and four students 
said it was very challenging; (5) what students are learning is 
very useful; (6) no one said that they would not recommend the 
course to fellow employees; and (7) all but one said that s/he 
would take another course if it were offered. 

The math midcourse critique helped the instructor to adapt his 
pace of instruction. It was a good tool for a quick, midcourse 
"check-in" with students. 

• Program Log. The program log was a good-enough idea 
that did not implement well. 

There was a need to document the rich anecdotal information 
about the program which circulated throughout the organization. 
Following the advice of the evaluation facilitator, the On-site 
Coordinator designated a special notebook as the program log and 
made it available to everyone in the class. He placed it on his 
desk ~ normally a busy place where people gather to talk ~ with 
the hope that students and others would write down their ideas 
and feelings about the program. 



Virtually no one used the log. No one used it because its 
purpose was not clear, and students (and others) were not being 
regularly encouraged in class and elsewhere to use it as a means 
for improving the program. This is not a criticism of the teachers 
or other staff. It does point out that the purpose for a collective 
journal needs to be continuously clarified by all staff and that 
staff need to model its uses and results. Without this, it becomes 
either intimidating or meaningless. 

Staff are figuring out how to best document the anecdotal 
information about the program that they do not want to lose. 
Using computers at work stations was being explored as an option. 

• Process Evaluation. These are the seventeen components of 
a quality program which the team reviewed, and the actions team 
members needed to take to bring the component into "quality 
range." 

Component #1: Articulate Philosophy of Program. 
No action 

Component #2: Develop and Implement Outreach and 
Recruitment Plan. 

Action: 

• More people want to enroll than the program can 
accommodate. There is a limit to the number of people who 
can be away from the machines at one time. The team 
needs to make a decision about the level of instruction the 
program should provide and who the program should target. 
Agreed: Target the program to people with the lowest skills 
and bring everyone up to basic mastery. Keep the schedule 
on shift, and then consider advanced classes. If the 
company becomes "more prosperous" it might be possible to 
get temps to cover during class time, but this is "easier said 
than done." 

Component #3: Provide Orientation 

The team agreed that the students are the "best ambassadors" 

but actions might include: 

• Eric talks to supervisors about the program. 

• Send message through electronic mail to sign up for GED 
and other messages about the program, too. 

• Put announcement about the GED program and the basic 
skills program in the newsletter. 
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Component #4: Establish Intake Procedures 

No action 



Component #5: Design and Implement Initial 

Assessment Activities 

Action: 

• Think about the assessment procedures that need to be in 
place for the next cycle. Does the education staff want to 
develop Individual Education Plans (lEPs) for each student? 
How can initial assessments and lEPs become part of a 
portfolio assessment? (Keep in mind that the program is 
respectful of people's fears of testing, and assessments 
shouldn't scare people away.) 

Component #6: Develop Curriculum 
Actions: 

• Move toward developing a more contextualized 
curriculum. 

• Remove some slang from the glossary and move toward 
more-generic English. 

• Document the curriculum. In Cycle 3, instructors should 
keep a "diary" of strategies they use in each class. 

Component #7: Select Appropriate Learning 

Arrangements 

Action: 

• Do not group more than three levels of students in one 
class. 

• Possible volunteer support to help teachers manage the 
range of skills students have. 

Component #8: Develop On-going Assessment 

Procedures 

Action: 

• Devise a way to document ongoing student progress ~ not 
just document at the beginning and the end of a course. 

• Revisit this issue in September and possibly invite Kate 
Camarra to provide technical assistance on developing a 
portfoUo assessment system. Pursue portfolio assessment 
whether Kate is available or not. 

Component #9: Ensure Support Services 
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No action 



Component #10: Staffing 
Action: 

Staffing is relative to how the program develops. Decisions 
need to be made about what the program should/could look 
like after federal funding ceases in December. For example: 

• Should the program serve a new population? 

• Should the program serve the current population at a new 
level? 

• Should computer literacy become part of the program? 
September is not too soon to think about December. The 
team suggested tliat it draw up a plan-plus-cost for a 
continuation program 

Component #11: Staff Training and Development 
Action: 

• Staff should access more of the training available through 
the Bureau of Adult Education 

Component #12: Program Planning, Evaluation, 

Monitoring and Management 

Actions: 

• Rename the "Language Arts" class. "Communication Skills" 
is a possible alternative. 

• Program policy is that if students sign up, effort should be 
made to come to class. Teachers might publish course 
outiines so that students can see what they're committing 
themselves to. 

• Add more students to the Planning and Evaluation Team. 
Add someone who is not a student so s/he can understand 
what program is about, and so that the team gets his/her 
perspective. 

• Start a "Program Log" ~ a big book or binder where 
everyone associated with the program and the company can 
write a word or more about what they feel/think about the 
program. (See memo to Fred about the "Program Log.") 

Component #13: Tracking and Reporting 
No Action: 

Component #14: Fiscal Management 
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No action (but questions about fiscal management are relative 
to how the program continues). 



Component #15: Facilities 

No action (although it is noted that the program is not 
handicapped accessible.) 

Component #16: Follow-up/Exit 

No action 

Component #17: The Business Education Partnership 

No action 



The Goals Evaluation Matrix 

The following Goals Evaluation Matrix presents the program 
goals along with the data-collection procedures used to collect 
information related to those goals. On the basis of the information 
collected from the sources just described, an assessment is made 
about whether the goals were achieved or not and, if they were 
achieved, to what extent were they achieved. An asterisk indicates 
that the assessment needs some explanation. The explanation is 
provided at the end of the Matrix. 
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Haartz Corporation Workplace Education Program 
Goals Evaluation Matrix 

• = This goal was evaluated 

A = This goal was achieved, as indicated by the related measure 

AP = This goal was achieved partially, as indicated by the related measure 

N A = This goal was not achieved 

* = See notes at end of Matrix 



INSTRUMENTS/PROCEDURES FOR 
DOCUMENTING AND MEASURING GOALS 



GOALS 






1 

^ ""\ 


C 

VJ 


>: 

'.-^ 


^- 


1 . Viewing the student as "customer," 
we want to offer them a high quality 
product 




• A 


• A 


• A 


• A 


• A 


2. To improve problem-solving 
and communication skills with 
students to encourage their 
coming forward with solutions 


•AP* 


• A 






• A 


• A 


3. To support and enhance Vertical/ 
Horizontal team concept * 1 




• A 






• A 




4 To establish within the culture 
of the Haartz Corporation a 
dynamically evolving philosophy 
of life-long learning and to 
create a bridge to other programs 
such as GED * 2 




. A 






• A 


• A 
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Notes on the Matrix 



* The pre-post survey did not document the kinds of gains in communication 
skills that were anticipated. As described, this was the result of student 
attitudes about being assessed on such a survey. Although the data were 
compromised, they still indicate that some communication skills were 
gained, if only through the process of making explicit with the instructors 
what the problems with the baseline data were. 

* 1 The team did not discuss this goal explicitly very much as the grant year 
progressed, but It is fundamental to the changes that the company is trying to 
implement. Workers in the focus group did not use the language of 
vertical/horizontal teams but they did, in fact, reference improved teamwork 
throughout their discussion. 

* 2 The most obvious indicator of the achievement of this goal is the 
expansion of the basic skills program to include an on-site GED program 
from which eight of twelve students received their GEDs after one course of 
instruction. 
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Preparing a strategy for reporting the findings 

Strategies for reporting evaluation results were targeted to 
specific audiences. There were several audiences for this evaluation. 
The team had identified these audiences early in the evaluation 
process, while still identifying goals and indicators. 

( 1) The first audience was the members of the Planning and 
Evaluation Team itself. They were the first to receive reports 
from the evaluation facilitator and others about the results of any 
data-gathering activity. These reports were usually oral, and 
were later written in memo or report form. This audience 
included the company President. 

(2) The second audience was other managers and employees in 
the company. They learned about the evaluation results 
throughout the year by word-of-mouth and reading articles or 
updates about the program in the Haartz Corporation newsletter. 

(3) The third audience was other programs in the Massachusetts 
Workplace Education Initiative. The Program Coordinator, On-site 
Coordinator, and Evaluation Facilitator attended several meetings 
and conferences where they discussed the progress of the 
evaluation. 

(4) The fourth audience was the program's two sponsoring 
government agencies: the U.S. Department of Education and the 
Massachusetts Department of Education. The Program Coordinator 
completed a final report of the evaluation for these two audiences, 
a product which has value for the other audiences, as well. 



Phase IV : Deciding what happens next 
Taking follow-up action 

The team continuously took follow-up action and improved its 
program throughout the grant year, based both on what it was 
learning through the process of planning its data-gathering activities 
and on the actual products of subsequent data collection and 
analysis. 
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The team learned many things that are the result of discussion 
and analysis of problems prior to the formal collection of data. For 
example, discussions about indicators of communication skills led to 
an understanding among team members that some supervisors did 
not support the program sufficiently, most likely due to lack of 
information and discussion about it. The On-site Coordinator decided 
to lead a discussion about the program at a supervisors' meeting. 

The actual collection of data ~ from the math midcourse 
critique, to the communication skills survey, focus group, and the 
process evaluation ~ also informed actions that the team took or 
might take to improve its program that have been discussed 
elsewhere in this case study. 

Now that the program is being run with company money, the 
team continues to meet and oversee planning and evaluation 
activities. Participants in the employee focus group suggested that 
focus groups be held every three months so that there is a regular 
forum for collective discussion about the program. They also 
suggested that supervisors meet in focus groups to discuss the 
program. Presumably, the team is considering these options. 



Evaluating the evaluation 

• Overall, the team was pleased with the evaluation process and 
results -- defining results broadly as the totality of what the team 
learned. 

• The Program Coordinator had been concerned that evaluation 
activities were taking too much time. (He was especially 
concerned about taking up too much supervisor and worker time 
in evaluation activities. This may have contributed to the delay of 
the supervisors' focus group.) However, at the end of the period 
in which the facilitator met regularly with the Haartz team, the 
Program Coordinator said that he understood the benefits of the 
process. 

• A survey was not the evaluation instrument most suited to the 
Haartz Corporation culture. However, the exercise of designing the 
survey, collecting the data, and analyzing it was invaluable. It 
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was a good "exercise" for the team, providing a platform from 
which it can evolve more meaningful data-collection techniques. 

• With hindsight, the team could have made a more concerted 
effort to complete the work which had been agreed on -- 
mastering SPSS and conducting a supervisor focus group. The 
facilitator could have urged the team more strongly to complete 
its work. 



IV. REFLECTIONS 

FaciUtator Laura Sperazi's comments: 

• This evaluation was made easy by having a team already in 
place when the facilitator began her work. Team members 
represented all the stakeholders and they conceived of 
themselves as a team that could - and should -- work for the 
benefit of the basic skills program. 

• The process evaluation sessions worked very well. They 
worked so well that we should have spent more time working out 
tlie actions that needed to be taken to improve quality services, 
and then documenting those actions and their results. We could 
have taken better advantage of the On-site Coordinator's 
knowledge of SPC processes and integrated them into the process 
evaluation of the basic skills program. We could have made the 
purpose, function, and operation of SPC processes and the process 
evaluation of the basic skills program more explicitly parallel. 

• The facilitator could have pressed the team to master SPSS and 
conduct a supervisor focus group -- work that was part of the 
loosely-defined contract between the team and the facilitator. But 
pressing these parts of the evaluation would have been awkward. 
The team worked very hard and did as much as it felt able to do. 
At the end, they were open to continuing evaluation activities on 
their own. If the facilitator had made the evaluation unpleasant 
by pressing for more work, the team may not have been so open 
to continuing on their own. 

• The evaluation plan could have better accessed the Haartz 
Corporation's "oral" culture and informal information streams 
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about the basic skills program. At Haartz, information about all 
aspects of work flows freely in conversation, and traditional ways 
of collecting information - like surveys -- can seem lifeless by 
comparison. (For example, the President of the company walks 
the floor each day and engages workers in informal conversation 
about how they and their families are doing.) The facilitator could 
have encouraged the team to think more creatively about 
alternative ways to collect information that would have accessed 
the company's oral culture. 

The worker focus group did access this source of information. 
It was an exciting event that helped to move the program 
forward. The program log had the potential to do the same. 
Presumably, the log will work better on computer than it did in 
"hard copy." 
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The HAARTZ Corporation ^ ^ 

Pre-test for Students Enrolled in Cycle 2 -h tcS/ Trr^ 
Focus on: Communication Skills at Work 

Your answers to the following questions will help the program managers to 
improve the program. Please read the questions carefully and think about your 
answers. This information is CONFIDENTIAL. We do not want to know who you are. 
We do want to know what you think. So, please tell us what vou reallv think! 

Please circle the box which best describes you now. If you would like to tell us 
something more that the box allows, write your comments in the space provided. If you 
need more space, use the back of the questionnaire. For example: 



1 . I help my children with their homework. 



not at all sometimes a lot 



always does not apply 



Comment:. 



1 . I am: 



male 



female 



2. I have worked at HAARTZ for: 

* Under six months 

* Between six months and a year 

* Between one and three years 

* Between three and five years 

* Between five and ten years 

* Over ten years 



3. What class did your take in the first cycle ? math 



language arts 



none 



4. What class are you taking new? math 

A -1- 



language arts 

1S2 



5. I speak up in meetings. 



□ 



not at all sometimes a lot always does not apply 



Comment: 



6. I say what's really on my mind in meetings. 



CZ 



□ 



not at all sometimes a lot always does not apply 



Comment: 



7. I use technical words when I need to. 



□ 



not at all sometimes a lot always does not apply 



Comment: 



8. I make suggestions about how to improve the way we work. 



□ 



not at all sometimes a lot always does not apply 



Comment: 
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9. I ask my supervisors questions when I need to. 



□ 



not at all sometimes a lot 



always does not apply 



Comment: 



10. I ask the people I work with questions when I need to. 



□ 



not at all sometimes a lot 



always does not apply 



Comment: 



* What do you ask about? Check the ones which are true for you. 

how the job gets done 

when new equipment will be installed 

suggestions to make work easier 

401 K 

Profit sharing 

Benefits, including health, dental, life insurance, sick leave, etc. 

Other things. Please tell us what they are. 



11. I talk to my supervisors with confidence. 



□ 



not at all sometimes a lot always does not apply 



Comment:. 



12. I let supervisors know that I have important skills, and that 1 am ready to use them. 



□ 



not at all sometimes a lot always does not apply 



Comment: 



13. My supervisors understand me when I talk to them. 



□ 



not at all sometimes a lot always does not apply 



Comment: 



14. The people I work with understand me when I talk to them. 



□ 



not at all sometimes a lot always does not apply 



Comment: 



1 5. i talk with my supervisor about ideas that improve quality. 



□ 



not at all sometimes a lot always does not apply 



Comment:. 



A -4 
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16. I talk with people I work with about ideas that improve quality. 



□ 



not at all sometimes 



a lot 



always does not apply 



Comment: 



*lf you feel that you do not know how to talk with your supervisors about quality, 
what would help you? Check what is true for you. 

Supervisors talk to you more 

Better instructions 

More information about why quality is important 

Other things that would help, Please tell us what they are. 



17. My supervisors let me know I have good ideas. 



□ 



not at all sometimes a lot always does not apply 



Comment: 



18. The people I work with let me know I have good ideas. 



□ 



not at all sometimes a lot always does not apply 



Comment: 
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19. I read job instructions. 



□ 



not at all sometimes a lot always does not apply 



Comment: 



20. I read safety procedures. 



□ 



not at all sometimes a lot always does not apply 



Comment: 



21. I read bulletin boards. 



□ 



not at all sometimes a lot always does not apply 



Comment: 



22. I work well with the other people on the floor. 



□ 



not at all sometimes a lot always does not apply 



Comment: 
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23. I offer help to my supervisors. 



□ 



not at all sometimes a lot always does not apply 



Comment: 



24. I offer help to people I work with. 



D 



not at all sometimes a lot always does not apply 



Comment: 



25. I teach new workers about how the workplace runs. 



□ 



not at all sometimes a lot always does not apply 



Comment: 



26. i use the information on the floor (i.e., computer print outs, memos, etc.) to improve 
how I do my work. 



□ 



not at all sometimes a lot always does not apply 



Comment:. 
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The HAARTZ Corporation 
Survey for Students Enrolled in Cycle 1 
Focus on: Communication Skills at Work 
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As you know, the basic skills program has been in operation for about seven months 
Program managers are interested in finding out what effects the program is having. Your 
answers to the following questions will help the program managers to improve the proqram in the 
next cycle. 

Please read the questions carefully and think about your answers. This information is 
CONFIDENTIAL. We do not want to know who you are. We do want to know what you think you 
have learned. So. please tell u.s what ym | really thinki 

The first time you answer the question, circle the box which best describes you before you 
attended your class. The second time you answer the question, circle the box which best 
describes you after you attended your class. If you would like to tell us something more than the 
box allows, write in the space provided. If you need more space, use the back of the 
questionnaire. For example: 



1 



not at all 



help my children with their homework. 
^ Before 



soroe- 
(imes 



a lot always 



does not 
apply 



not at all 




some- 
times 



does not 
apply 



1 . i am: male female 

2. I have worked at HAARTZ for: 

*Under six months 
*Between six months and a year 
*Between one and three years 
*Betv/een three and five years 
*Between five and ten years 
*Over ten years 

3. What class did you take in the first cycle? math language arts 

4. What class are you taking now? math language arts 
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5. I speak up in meetings. 



Before 



not at all some- a lot always does not 
times spply 

Comment: 



not at all some- 
times 



After 

a lot always 



does not 
apply 



6. I say what's really on my mind in meetings. 



Before 
Mil 



not at all some- a lot always does not 
times apply 

Comment: 



not at all some- 
times 



After 

■■I] mam 

a lot always does not 
apply 



7. I use technical words when I need to. 



Before 



not at all some- a lot always does not 
times apply 

Comment: 



not at ail some- 
times 



After 

a lot always 



does not 
apply 



8. I make suggestions about how to improve the way we work. 

nzz: MHZ 

not at all some- 
times 

Comment: 



Before 

Bi] Ml 

a lot always 



does not 
apply 



not at all some- 
times 



After 

a lot always does not 
apply 



Q -2 



IerIc 



ISO 



9. I ask my supervisors questions when i need to. 



not at all some- 
times 




r><:i 

does not 
apply 



not at all 



some- 
times 



After 



a lot always 



does not 
apply 



Comment: 



10. I ask the people I work with questions when I need to. 

After 

(=] WZU mmi H 

not at all some- a lot always does not not at all some- a lot always does not 

times apply . times apply 

Comment:^ 




*What do you ask about? Check ones which are true for you. 

how the job gets done 

when new equipment wili be installed 

suggestions to make work easier 

401 K 

Profit sharing 

Benefits, including health, dental, life insurance, sick leave, etc. 

Other things. Please tell us what they are. 



11. I talk to my supervisors with confidence. 

After 

not at all some- a lot always does not 
times apply 

Comment: 




not at all some- a lot always does not 
times apply 
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12. 1 let supervisors know that I have important skills, and that I am ready to use them. 



Before 



not at all some- 
times 



a lot 



always does not 
apply 



not at all some- 
times 



Comment: 



After 

a lot always 



does not 
apply 



13. My supervisors understand me when 1 talk to them. 



Before 



After 

mm 



3 



not at all some- a lot always does not 
times apply 



not at all * some- a lot always does not 
times apply 



Comment:. 



14. The people I work with understand me when I talk to them. 



C 



Before 
Hi] 



After 



not at all some- a lot always does not 
times apply 



not at all some- a lot always does not 
times apply 



Comment:. 



1 5. I talk with my supervisors about ideas that improve quality. 



Before 



not at all some- a lot always does not 
times apply 

Comment: 



After 



not at all some- a lot always does not 
times apply 
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16. I talk with people I work with about ideas that improve quality, such as, better instructions for 
Standard Operating Procedures. 



Before 



not at all some- a lot always does not 



not at all 



times 



apply 



some- 
limes 



After 

wami mmm 

a lot always does not 
apply 



Comment: 



17. My supervisors let me know I have good ideas. 



Before 



not at all some- a lot always does not 
times apply 

Comment: 



not at all some- 
times 



After 

a lot always does not 
apply 



18. The people I work with let me know I have good ideas. 



L 



Before 



not at all some- a lot always does not 
times apply 

Comment: 



] i: 



not at all some- 
times 



After 

■■■ 

a lot always does not 
apply 



19. I read job instructions. 



Before 



wm 1^^ 

not at all some- a lot always does not 
times apply 

Comment: 



not at all 



some- 
times 



After 

■■1] mam-^ f^<n 

a lot always does not 
apply 



6 -5- 
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20. I read safety procedures. 

Before ^ ^ After 

not at all some- a lot always does not not at all some- a lot always does not 

times apply times apply 

Comment: 



21. I read bulletin boards. 

Before 

not at all some- a lot always does not 
times apply 

Comment: 



After 

not at all ' some- a lot always does not 
times apply 



22. I work well with the other people on the floor. 

Before After 



not at all some- a lot always does not not at all oome- a lot always does not 

times apply times apply 

Comment: 



23. I offer help to my supervisors. 



not at all some- 
times 



Before 

■■■ 

a lot always does not 
apply 



Comment:. 



not at all 



some- 
times 




always 



does not 
apply 
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24. I offer help to people I work with. 



not at all some- 
times 

Comment: 




does not 
apply 



not at all some- 
times 




does not 
apply 



25. I teach new workers about how the workplace runs. 




not at all some- a lot always does not 
times apply 

Comment:^ 



not at ail some- 
times 




does not 
apply 



26. I use the information on the floor (i.e., computer print outs, memos, etc.) to improve how 1 do 
my work. 



r 



Before 



not at all some- a lot always does not 
times apply 

Comment: 



After 



not at all some- a lot always does not 
times , apply 
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OFFICE OF THE SUPERINTENDENT 



Arthur Goodall EH rector 
Faculty Ebctcrrwhip Pit}^ini 




May 24, 1993 



TO: Laura Sperazi 
FROM: Art Goodall 

RE: Haartz Corporation - Draft Survey for Students Enrolled in Cycle 1, 5/5/93 
Focus on: Communication Skills at Work 

On this day both Rodney Goguen and George Pickles were asked to respond to this 
survey with Art Goodall serving as the survey facilitator. The Survey was given in 
a comfortable, quiet, welHit space identified as the Lab Conference Room for 
Rodney Goguen and the Building 14 classroom for George Pickles, both located at 
The Haartz Corporation. 

It was explained to Messers Goguen and Pickles that they had been selected from a 
group of student workers who had completed Cycle 1 of The Haartz Workplace 
Education Program. It was further noted that this was not a test that would be 
graded and that there were "no wrong anwsers". 

Prior to actually starting to answer the questions, Mr. Goguen noted that Workplace 
Education was a very valuable program at Haartz and that he had personally 
profited from this experience. He specifically singled out Mr. Kenneth Russell as a 
very able instructor in Mathematics and that he hoped the Program would grow and 
include an even broader variety of people. 

Mr. Pickles felt that the Language Arts Program, instructed by Meaghan Mahoney 
offered considerable individualized instruction. He also felt that good opportunity 
had be^n provided to practice a variety of communication skills. Meaghan 
Mahoney was spoken of with warm praise for the variety of opportunity which she 
provided to her classes. 

George Pickles noted that the Language Arts Class opened lines of communications 
between workers which, prior to this class, were closed. A comraderie or esprit de 
corps was built to a level, which heretofor had not been as evident. Mr. Pickles 
noted that Ms. Mahoney worked with his group offering a variety of techniques 
including the debate of issues, definition of words and some grammar, many times 
within the context of a game. Mr. Pickles felt that many members of the class 
benefited from the stimulation of these activities and discussions bringing such 
people out as a result of a boost in their self confidence. When asked, George Pickles 



Laura Sperazi 
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March 24, 1993 



felt that some members of the company who had not participated in either Cycles 1 
or 2 were being positively impacted by the presence of the Program within the 
workplace. He and others are being asked questions about the Program from those 
individuals who are considering signing-up for Cycle 3. 

With respect to the Survey Instrument, Mr. Goguen made the following points: 

• The first two sentences in paragraph 3 are confusing. One must read these 
sentences twice and then read the rest of the paragraph to make complete sense 
of the direcitons. 

• With respect to Questions 6, 7, and 8, because Mr. Goguen has always spoken up 
in meetings, the math class experience did not result in significant change. 

• With respect to Questions 12 and 15, because Mr. Goguen has recently been 
promoted to a new position, he has been anxious to let supervisors know about 
his skills and his confidence in handling the challenges of the new assignment. 

• With additional commentary on Question 15, Mr. Goguen now discusses quality 
to a greater extent in his advanced position and is looking for additional time to 
engage in more of these discussions. 

• With respect to Question 26, Mr. Goguen suggested that we incorporate examples 
of what is meant by "information on the floor". 

The Survey was completed in approximately 20 minutes. Please find enclosed 14 
copies of The Haartz Corporation Pre-test for Students Enrolled in Cycle 2. 

With respect to the Survey Instrument, Mr. Pickles questioned Item #16 especially 
the phrase "Better Instructions". He suggested that the item could be strengthened 
by extending the phrase to include, "Better Instructions for Standard Operating 
Procedures". 
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Haartz Corporation Workplace Education Evaluation 

July 22. 1993 

Questions for Worker Focus Group 

1. What are the two most important things that you've learned in your 
class? Why are these things so important? 

2. Are communication skills necessary to do good work at Haartz? 

What do the words "communication skills" mean to you? 

- Speaking 

- Reading 

- Writing 

- Listening 

- Other? 



3. Does being in class help you to improve your communication skills? 
Why/How? 

• Please give us an example of how being in class helps to improve your 
communication skills. Be specific. 

For example, does being in class help you to: (Can refer to indicators list here) 

- read job instructions 

- talk to your supervisors more 

- read bulletin boards 

4. Have you changed as a result of being in class? How have you 
changed? 

• Have you seen others change? How have they changed? 

• Has the organization changed as a result of sponsoring classes? How has it 
changed? Have supervisors changed? If so, how? 

• Do you know that federal support for the program will end in January? 

• Would you like the company to continue classes beyond January? 

• What would you like to the program to include? 

5. What things does the teacher do in class that help you learn the 
most? {Talk to you, give you assigments, facilitate group discussion?) 

• What would you like the teacher to do more of? 

• is there anything else you want included in your class? (Videos, audio tapes, 
etc?) 
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Math Midcourse Critique 

1 . How do you feel about the course at this point? 

a. Very happy 

b. Happy 

c. Satisfied 

d. Dissatisfied 

e. Unhappy 

2. Is the course nnoving 

a. Much too fast 

b. Too fast 

c. Just about right 

d. Too slowly 

e. Much too slowly 

3. Does the instructor explain things 

a. So I understand very clearly 

b. So I understand clearly 

c. So I understand 

d. So I'm not sure I understand 

e. So I don't understand or am confused 

4. Does the instructor make things 

a. Very interesting 

b. Interesting 

c. OK 

d. Not very interesting 

e. Boring 

5. Is the material 

a. Very challenging 

b. Challenging 

c. OK 

d. Not very challenging 

e. Boring 

6. Is what you are learning 

a. Very useful 

b. Useful 

c. OK 

d. Not very useful 

e. Useless 

7. Would you recommend this course to your fellow employees? 

a. Absolutely b.Yes c. Maybe d. I doubt it e. No 

8. Would you be interested in taking another course if it is offered? 

a. Absolutely b.Yes c. Maybe d. I doubt it e. No 

COMMENTS: 
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TEAM EVALUATION 
AT THE NORTON COMPANY : 
A CASE STUDY FROM A WORKPLACE EDUCATION 

PROGRAM 



Prepared 
by 

Laura Sperazi 



One of seven case studies in 
" Team Evaluation : 
Case Studies from Seven Workplace Education Programs," 
a report prepared in June 1994 
by Laura Sperazi and Paul Jtirmo 
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THE NORTON COMPANY 
Worcester, Massachusetts 



I. PROGRAM BACKGROUND 
Program history 

The Norton Company, founded in 1885, is a world-wide 
manufacturer of grinding wheels, diamond cutting products, and 
sand paper. It sells its products primarily to industrial customers. 
The Companie St. Gobain, a large French corporation which began 
producing glass in the seventeenth century and now produces 
insulation, pipes, silicon wafers, and other such products, bought out 
the Norton Company in September of 1990. The buy-out prompted 
some of the organizational changes which the Norton Company is still 
undergoing ~ changes which include a new emphasis on quality, on- 
time production, and team-based management. 

The Norton Company employs almost 16,000 people world- 
wide. The Bonded Abrasives Division, for which the original Norton 
Company job-related math skills course was designed, manufactures 
grinding wheels and employs about 1,200 people in Worcester, 
Massachusetts. An explicit goal of the Norton Basic Skills Program is 
to facilitate the organizational changes that are intended to enhance 
the company's competitiveness. 

However, there is a bias against organizational change "buzz 
words" iii this culturally conservative company. Managers do not 
loosely use "TQM" or "continuous improvement" or other fashionable 
phrases the way other companies might because, according to one 
manager, "there is so much diversity in what those phrases mean." 
Manager of Human Resources struggle deeply with defining clearly 
the essential changes in the organization of work that are needed to 
support their new business strategy. They struggle to define and 
provide the support, which employv^es need to perform in accordance 
with those changes. And members of the Norton Basic Skills 
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Committeel struggle to define the appropriate goals and measures 
for their Basic Skills Program so that there will be symmetry among 
the goals of the business , the goals of the training program , and the 
goals of the evaluation . 

The "bottom line" of change at the Norton Company is that 
"there is some reorganization of work and an associated transfer of 
power and decision making to more people." As one Manager of 
Human Resources said: "We have to re-engineer the way we do 
business. Htmian potential needs to be developed individually and 
collectively. Without a re-engineered human component, any other 
process wUl not succeed." The company ~ and the training 
department in particular ~ need to move workers to "where the 
work is." Bonded abrasives is a "ver>' fickle" business; the work 
changes rapidly. Therefore, the company needs to move employees 
in the direction of the demand for work without having the pace of 
change frighten ~ and perhaps incapacitate ~ them. . 



Program goals and components 

The introduction of organizational changes and related 
computerized technologies clarified the need to provide math 
instruction to some employees. Within a couple of years of providing 
math instruction (which began in 1990), the need for additional 
instruction in language arts, blueprint reading, and measurement 
was clarified. The opportunity to conduct a formal evaluation of the 
Basic Skills Program further clarified the need to define measurable 
goals and indicators for the program. (See "Clarifying program goals 
and the information to look for," below.) 



1 The Norton Company Bonded Abrasives Basic Skills Committee is 
virtually synymous with a Planning and Evaluation Team. The Basic 
Skills Committee is composed of representatives from all the 
stakeholder groups, including program participants. The Basic Skills 
Committee oversees the planning and operation of the basic skills 
program and was involved in the early and middle phases of 
conducting the evaluation. In practice, a few members of the Basic 
Skills Committee have taken responsibility for carrying out the last 
phase of work for the evaluation. Throughout this case study, Basic 
Skills Committee and evaluation team are used interchangeably. 
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As of November 1993 a total of ten classes were offered over 
three shift periods. The classes included: three math; three 
language; two computer-enhanced blueprint reading; one 
introduction to measurement; and one machine maintenance class 
taught by an employee on the Basic Skills Committee. (Teaching the 
class was this employee's idea and grew out of his participation on 
the Basic Skills Committee) 

The company provides 50% release time to all Basic Skills 
Program participants. About one hundred thirty students have 
completed a course of instruction in the program. There are about 
100 participants enrolled in the current course offerings. (This 
includes participants who have completed other courses in the Basic 
Skills Program and are continuing to receive instruction.) 

The Workplace Education Program within the Center for Life- 
Long Learning at Quinsigamond Community CoUege in Worcester, 
Massachusetts is the learning provider for the Norton program. 



II. BACKGROUND ON THE TEAM 
History of the team 

The team-based evaluation of the Norton Basic Skills Program 
began in late April 1992. At that time the Norton Basic Skills 
Committee had already been meeting regularly every two weeks for 
eight months. At that time, the Norton program was funded through 
the Massachusetts Workplace Education Initiative (MWEI), Cycle 5. 

The presence of a committed Basic Skills Committee in the 
early stages of the Norton program was the result of two factors. 

First, the Workplace Education Program Coordinator at the 
Center for Life-long Learning at Quinsigamond Community College 
had participated in the MWEI's evaluation of its fourth cycle of 
sponsored programs. This evaluation used evaluation teams to 
conduct their own evaluations for the first time in Massachusetts. 
The benefits of that experience encouraged her to replicate the team 
evaluation model in her new program from the beginning. 
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Second, the Manager of Human Resources was accustomed to 
working in teams and immediately saw the value of regular 
communication about the development and evaluation of the Basic 
Skills Program. The evaluation facilitator met with the Norton Basic 
Skills Committee (this is what the team called itself) first under the 
auspices of the MWEI - from April through June ~ and then under 
the auspices of the National Institute for Literacy research project - 
from December 1992 through March 1994. 

The members of the Basic Skills Committee were ready and 
eager to address questions about the outcomes they could expect 
from their program. The team focused almost exclusively on 
organizational outcomes, and this focus was a function of three 
important factors: 

(1) the need to understand the full dimension of the 
organizational changes the company was/is imdergoing; 

(2) the imderstanding that these organizational changes will be 
achieved largely through educating and retraining employees; and 

(3) the intention to link organizational changes directly to the 
training and education programs which are intended to improve 
the skills of the workforce. 

It is important to note, however, that despite the focus on 
organizational outcomes, the team supported the achievement of 
worker's personal goals through enrollment in the Basic Skills 
Program. Achieving personal goals would simply not be the focus of 
the evaluation. 

Stakeholders represented on the team 

Committee members included: 

• Donna Holden , Administrative Coordinator, Training Department 

• Bethany Brockmeyer, Educational Coordinator for the North 
American Abrasives Group 

• Ray Bull, Training Coordinator 

• Brian Eyles, Program Participant, Former Floor Operator (having 
suffered an injury at work, he now works in the administrative 
office) 
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• Brad Field, Systems and Sourced Product Manager for 
Superabrasives Division 

• Jack Gagnon, Floor Operator, Program Participant 

• Beit Galarza, Floor Operator, Program Participant 

• Joseph Javorski, Manager of Human Resources, Precision 
Business 

• James Litwinowich, Manager of Quality 

• Joseph Passeri, On-site Program Coordinator and Instructor 

• Kathy Rentsch, Workplace Education Program Coordinator, 
Center for life-long Learning, Quinsigamond Conmiimity College 

• Mark Stacey, Manager of Human Resources 

• Cal Wilson, Departmental Supervisor 

The composition of the Bacic Skills Committee changed 
considerably during the fifteen months of the NIFL-sponsored 
evaluation. The people named above were all part of the group for a 
period of time. However, a core group of people have provided 
continuity in the committee for this period. These people include 
Bethany Brockmeyer, Joseph Javorski, and Joseph Passeri. Kathy 
Rentsch was a regular member of the team until May 1993 when she 
left her job as Workplace Education Program Coordinator at 
Quinsigamond Commxmity College. 



III. THE PROCESS WHICH THE TEAM WENT THROUGH 

Summary: Between April and July 1992, the evaluation facilitator 
met with the Norton Basic Skills Committee three times. Between 
December 1992 and April 1994, the evaluation facilitator met with 
the Norton Basic Skills Committee eight times. 

Four of the eight meetings were held on-site at the Norton 
Company with trainers for a computer data-base program called 
Statistical Package for the Social Sciences (SPSS). (Two of these 
meetings were day-long; two were half-day; all trainings were 
conducted for the Norton team exclusively.) Steven Andrews and 
Alice Oberfield Andrews conducted the SPSS training and were 
available for telephone consultation as needed. 

One of the meetings was held with Donald Cichon, of Donald 
Cichon Consultants, who advised on technical aspects of survey 
development. Donald Cichon was also available by phone as needed. 
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In addition to these eight meetings with the evaluation facilitator, 
Steven Andrews met once by himself with selected members of the 
Basic Skills Committee to advise on coding survey data. 

The Basic Skills Committee also met with the facilitator, SPSS 
trainers, and the evaluation teams from Bull Worldwide Information 
Systems and the Haartz Corporation once in early February 1993 for 
an introduction to SPSS and an overview of activities of the three 
evaluation teams. This maices a total of thirteen meetings and 
trainings which the Norton team participated in. 

During these meetings the team: 

• clarified its goals for the evaluation. 

• specified one primary indicator of success for its Basic Skills 
Program ~ improved employee involvement in the 
organization. 

• expanded the primary indicator to include involvement on 
particular jobs as well as in the organization as a whole. 

• developed several formats for a survey focused on employee 
involvement in the organization and on the job. 

• collected data from approximately 160 program participants, 
including maintenance data on approximately 30 participants. 

« coded the data according to SPSS requirements. 

• programmed the data. 

• attempted to run the data for preliminary analysis. 

• revised its evaluation plan. 

Only the most preliminary analyses of the data had been run at 
the time of the formal conclusion of the evaluation. There are 
several interconnecting explanations for this. They include: 

(1) Typing the coded data onto a computer was a more 
complicated task for a novice team to manage than either the 
team or the facilitator had imagined. It was more complicated 
because support staff to whom this task had been delegated had 
to learn a new data-entry format and because the data which the 
team had generated were voluminous. The SPSS format requires 
some study and it was time-consuming and tedious to type coded 
data for hundreds of variables and multiple versions of surveys 
onto a computer. 
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(2) Turnover in support staff further delayed the completion of 
this task. New support staff had many responsibilities other than 
finishing the computer work for what was probably perceived as 
a marginal project. This new person had no continuity with the 
evaluation and therefore no personal investment in completing it. 

(3) A job change for one key member of the team resulted in his 
spending less time thinking about and working with the 
evaluation project. 

(4) The SPSS consultants and evaluation facilitator were 
committed to the idea that team members should learn to run the 
SPSS program themselves, and it did not occur to them to contract 
out the input and analyses when it was probably appropriate to 
do so. The SPSS consultants could have completed the input and 
preliminary analyses themselves, an.d then advised the team on 
how to proceed on their own, with continued guidance as needed. 

(5) Furthermore, as the length of time between data collection and 
analysis grew, a new perspective on how evaluation might be 
conducted in a more decentralized manner was put forth by a key 
member of the team. This cast doubt on whether this evaluation 
should be concluded according to the original plan. 

Despite these drawbacks, work on the evaluation has been very 
valuable. Work on the evaluation has provided a continuous forum 
for key stakeholders to clarify what is important to know about what 
their program is achieving. One key member of the team, the 
Educational Coordinator for the North American Abrasives Group, 
has taken the initiative to lead the evaluation through the analysis 
stage. She is motivated to do so because she will be a resource in 
basic skills program development to as many as twelve to sixteen 
other Norton plants in the United States. She would like an effective 
evaluation model to share with others. The evaluation of the Basic 
Skills Program at Bonded Abrasives in Worcester gives her the 
opportunity to iron out snags in the model before replicating it. The 
key elements of the plan to continue the evaluation at Bonded 
Abrasives are described below in "Taking follow-up action." 
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Phase 1: Preparing the team 



Initial preparations 

As described above, the Norton Basic Skills Committee began 
meeting in September 1991 under the auspices of the Massachusetts 
Workplace Education Initiative. When the evaluation facilitator 
began to work with the team in April 1992, she did not have to 
persuade Basic Skills Committee members about the value of team- 
based evaluation. Committee members were quite open to 
considering evaluation activities as part of their work. The MWEI 
had prepared them for this. However, only a few members had any 
formal experience with evaluation. The facilitator introduced the 
purposes and methods of workplace education program evaluations 
and provided a forum in which questions related to evaluation could 
be addressed. The team's experience with the faciUtator was positive 
enough that they accepted her offer of continued facilitation imder 
the auspices of the National Institute for Literacy. 

Organizing the evaluation team 

Program staff took responsibiUty for organizing the evaluation 
team. See list of members, above. 



Phase II : Planning the evaluation 

Clarifying program goals and the information to look 
for 

This section explains how the team decided to focus on 
"employee involvement" as the central indicator of success of its 
Basic Skills Program, and how new dimensions of that indicator 
evolved over time. It is useful to understand the process which the 
team went through to decide on this particular focus. 

The team began by defining outcomes for the Basic Skills 
Program^ in the broadest of organizational terms: "creating a flexible 



2 When the evaluation was begun, the Basic Skills Program was 
called the Job-Related Math Program. Only math instruction was 
offered. It soon evolved into the Basic Skills Program where more 
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workforce." The team then specified some of the skills employees 
need in order to perform many jobs "flexibly." These new skills 
include math and measurement; communication; problem solving; 
and tmderstanding how the business cycle works -- lead time, on- 
time delivery, etc. 

The team got more specific about possible "quality outcomes" 
for the Basic Skills Program when it addressed the question of 
improvement of individual -- as opposed to team -- productivity. 
The team questioned whether it is possible to measure change in 
individual productivity based on participation in the Basic Skills 
Program. This was because work at Norton is conducted in teams, 
and that the organization will become only more team-oriented. 

In an attempt to deal with this question of measuring 
individual employee productivity, the team identified "better 
recording on SPC" as a desired outcome of the job-related math 
program. Properly defmed this would, in fact, be an effective 
measure of individual improvement in productivity. The evaluation 
team defined better recording on SPC in two ways: 

( 1) How quickly are employees identifying a process as out of 
control or tending to be out of control; and once the process is 
identified as out of control, how quickly can employees come to a 
solution for cause/variation? 

(2) How effectively can employees investigate the source of the 
problem and rework what needs to be reworked? 

At this point it became clear that better recording on SPC ~ as 
weU as all the other broad organizational outcomes which the team 
identified ~ are not the desired outcomes of the Basic Skills Program 
alone, but rather of the whole education and training effort. Put 
another way, a job-related Basic Skills Program cannot shoulder the 
burden of bringing about broad-based organizational change, or of 
improved use of SPC. 



than math instruction was offered. For the sake of clarity, 
throughout the rest of this case study we refer to the Basic Skills 
Program, unless it is appropriate to specify the Job-Related Math 
Program. 
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However, it does have a specific role to play alongside other 
education and training programs. That role is supporting the 
development of a culture of continuous product improvement 
through education and training, and of helping employees to 'bridge" 
effectively to other education and training programs within the 
company - programs in which they could not participate unless they 
had the benefit of basic skills instruction. Indicators of the success of 
the Basic Skills Program should then include how well the program 
helps employees to bridge to other programs, and on how class 
instruction fosters an environment of continuous improvement 
through education. 

By the end of the second team meeting with the evaluation 
facilitator, two important conceptual shifts occurred: 

(1) The team understood that participatory instructional methods 
in the classroom promote the very skills that the company wants 
to teach all its employees: creative problem solving, taking 
initiative, collaborating with others, etc. This means that 
employees who are enrolled in a basic skills course have an 
advantage over others. They receive extra training in the new 
skills which the company is promoting. One team member put it 
this way: "Instead of seeing education separate from 
manufacturing, we now see the integration. Job-related basic 
skills can work exponentially to change the organization. The 
instructor in the classroom is like a group leader. There is no wall 
that separates one process from another." 

(2) The team agreed to focus on "employee involvement" as the 
indicator of the success of their workplace Basic Skills Program. 
This desired outcome was the same outcome that other training 
efforts were trying to bring about. It was the outcome that 
seemed to capture the essence of the organizational change that 
the company was trying to bring about. At this time, the team 
defined employee involvement relative to the organization, not 
relative to job performance. Involvement in the organization 
might include -- among -^ther things - volunteering or being 
asked to take the lead role in a project; volunteering or being 
asked to make a presentation to supervisors; volunteering or 
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being asked to lead a company tour; continuing education; and 
bidding for better jobs. (See Appendix A.3) 

The following list details the goals which the team pursued in 
its early meetings, ending with improved employee involvement. 

• Flexibility of the workforce. Employees are learning new skills 
to perform more jobs. These new skills include: 

~ math and measurement 

- communication 
~ problem solving 

~ understanding how the business cycle works -- lead time, 
on-time delivery, etc. 

• Transferability of skills from class to work. 

• Bidding for better jobs 

• Individual productivity improvement. 

• Increase in employee self confidence. 

• Retention. 

• Decrease the number of rejects, "increase productivity" - 

• Better recording on SPC. 

- How quickly are employees identifying a process as out of 
control or tending to be out of control; and once the process is 
identified as out of control, how quickly can employees come 
to a solution for cause/variation? 

- How effectively can employees investigate the source of the 
problem and rework what needs to be reworked? 

• Employee involvement 

~ take the lead role in a project 

- continuing education 

By the third meeting, the team was further specifying the 
indicators of improved employee involvement in the organization, 
and drafting sample questions for a questionnaire which would be 
given to all participants in the Basic Skills Program. The team 
anticipated that the results of a questionnaire administered before 
and after enrollment would demonstrate improved employee 



^ Appendix A is the final draft of the survey instrument which 
focuses on employee involvement. It contains the original questions 
which focus on involvement in the organization, but it also contains 
new questions which focus on involvement on the job. The following 
sections of the case study describe how the focus evolved from 
involvement in the organization to involvement on the job. 
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involvement for most program participants. The team also 
considered identifying a control group so that the evaluation could 
attribute improved employee involvement to participation in the 
Basic Skills Program and not to other factors. 

With documented improvements in employee involvement 
resulting from participation in the Basic Skills Program in hand, the 
training program leadership would be in a strong position to argue to 
company managers that this type of training moves the company 
toward achieving its desired organizational change. The team 
completed the first questionnaire and used it to guide pre- 
enrollment interviews with 100 participants in September 1992. 

When the facilitator began to work with the team again in 
December 1992, about 60 additional questionnaires had been 
completed. 

However, in the winter of 1992-3, some issues concerning the 
administration of the survey, the content focus of the evaluation, the 
use of a control group, and the overall amount of work that the 
evaluation was requiring began to surface. These issues surfaced 
and resurfaced throughout 1993 in the following ways. 

• S trengths and limitations of the interview format 

The team had decided to gather the survey data in an 
interview between the program participant and the On-site 
Coordinator. These interviews took about an hour each. While the 
team felt committed to the quality of the information they were 
getting through this interaction, they also felt that the time being 
spent was excessive. One hundred interviews meant one hundred 
hours just gathering information. The time required to collect 
information sometimes led to information not being collected on 
time. If there were other things that the On-Site Coordinator had 
to do to keep the program going, those things would get priority. 
Data collection would take a back seat. 

• Options for data collection 

The goal was to minimize interview time and not completely 
sacrifice the interaction which enriched the participants' answers. 
One strategy was to have teachers administer the survey in class 
during a regular class period. All participants would answer the 
survey at the same time, thereby saving many hours. The teacher 
could make the process interactive. S/he could move through the 
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questions one by one, clarifying potentially difficult concepts, and 
being a resource for anyone who wants to ask a question. 

In early February, the team decided that it was willing to 
invest time in interviews (as distinct from group data gathering in 
class) as long as they had confidence in the information that the 
instrument was gathering. They decided to stay with the content 
focus on employee involvement, even though "the measure may 
not be 100%." They suggested paring down the interview form 
and triangulating information two or three different ways in one 
section - as a control on the kind of data that was being collected. 
They also decided to gather complete demographic information on 
participar.ts. 

• To have a control group -- or not 

At the same time, the team was considering advice from the 
SPSS trainers about selecting a control group. The trainer 
described three options for selecting a control group: 

~ randomly select a control group and test it at pre-post 
- randomly select a control group and test it at post only 
-- match each member of the experimental group with a 
control as close to age, numbers of years in company, etc. as 
possible. (See Appendix B) 

How to manage data collection with a conOrol group when the 
data collection for the experimental group was already 
burdensome was a problem which the Manager of Human 
Resources struggled with. Of all the team members he believed 
most strongly that the design of the evaluation would be 
compromised if there were not control group. 

The evaluation facilitator argued that there was value in 
descriptive data on the experimental group alone - especially at 
this stage of the evaluation process. But it is likely tiiat the 
Manager of Human Resources had a more-realistic view of what 
his managers would consider persuasive evidence of the benefits 
of the Basic Skills Program. The team never did include a control 
group in the evaluation activities because of the extent of the 
work required. 

• Catching UP with SPSS ti-aining 

In late January, in preparation for the first training in SPSS, the 
Manager of Human Resources installed the tutorial program and 
went through it to get a basic understanding of the system. In 
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mid-February the trainers offered their first training in SPSS to 
four members of the evaluation team. The trainers were not 
accustomed to teaching people whose knowledge of computers 
and statistics was limited and the result was a training that 
overwhelmed more than it instructed. 

One of the trainers returned to Norton to provide a more- 
tailored, hands-on training. But the team continued to struggle to 
keep the coding and analysis process on track. This was the result 
of infrequent on-site technical assistance sessions, job changes for 
key people on the team, and the enormous amount of data that 
had already been coUected, was stiU being collected, and needed 
to be coded. 

• Expanding the content focus of the evaluation 

The Manager of Human Resources had had some doubts about 
the content focus of the survey. Feedback from Donald Cichon, the 
measurement consultant, in early March clarified a gap in the 
content focus. The measurement consultant suggested that 
employee involvement in the org anization hP expanded to include 
employee involvement on the ioh . Specifically, this would mean 
modifying the survey questions to include items that assess if and 
how program participants are "more involved" at their jobs ~ 
collaborating more, taking initiative, problem solving, etc. These 
items come closer to assessing job performance than the original 
items. 

The Manager of Human Resources supported the redesign of 
the survey. He supported the amount of time that would have to 
be spent in the redesign and considered it "the cost of this kind of 
decision." He was interested in exploring further the "describable 
behaviors that support the business strategy" and in finding a 
way to show the company managers more specifically what they 
are getting from the Basic Skills Program. 

Employee members of the team suggested a process for 
identifying which elements of employee involvement on the job 
should be included in the survey. They did not claim to know 
exactly which elements should be included. They did suggest that 
if a few employees answered the following questions the team 
would be better able to define the component parts of improved 
employee involvement on the job: 

~ Do I want to learn other jobs? Which ones? Why? 
- How many wheels a day do I let go that I shouldn't? 
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- What can I do to help to improve quality and generally 
support the business, and why should I do it? 

The team also explored another option for defining the 
component parts of improved employee involvement on the job. 
That option was to train supervisors to document in an 
ethnographic way what employee involvement on the job is. This 
option would include supervisors in defining the evaluation 
criteria - something that, along with employee input, could only 
strengthen the program and validate the items. 

These options for involving employees and supervisors in the 
process of defining indicators for improved employee involvement 
on the job were exciting. Perhaps because these options are time 
consuming and build a new layer of data collection into the 
process, the team moved in another direction instead. 

The team opted to see how the Basic Skills Program evaluation 
might tap into or build on the new company-wide performance 
appraisal process and categories. Underneath this option for 
expanding the survey content through overlap with the 
performance appraisal process lay the question: should the team 
be looking at performance instead of involvement? There may 
still be some difference of opinion among team members about 
the answer to this question. 

• Using performance appraisal criteria 

The measurement expert reviewed the performance appraisal 
criteria and gave the team feedback. He suggested that the team 
use the new performance review as a framework for new survey 
questions but that they break down categories further. He also 
assured the team that the five question areas which the team 
adapted from the performance appraisal, and the three levels of 
information asked for within each area, were conceptually elegant 

The five question areas are: improvement in your work area; 
morale in your work area, communication between shifts; safety; 
work flow. The three levels of information are: awareness of 
problems; action needed to correct the situation; does employee 
initiate some action. These new questions are correlated to the 
Performance Evaluation so that, conceivably, information from one 
system can supplement information from another. (See 
Appendices C, D, E and F.)) 




Phase III: Gathering, organizing, analyzing, and 
reporting the information 

Gathering and organizing the data 

The Process Fvaluation: 

In May of 1993, the facilitator convened a special three-hour 
meeting of the committee to address the question: What needs to 
be in place in the program in order to enhance the likelihood of 
achieving the goals? In this meeting team members reviewed a 
set of indicators established by the Massachusetts Workplace 
Education Initiative for each component of a quality workplace 
education program and assessed their program against those 
standards. They judged whether their program was within 
quality range for each indicator or whether an action needed to be 
taken to bring the component into quality range. (See Appendix 
G.) 

The survey: 

The Norton Company evaluation focused at first on the 
involvement in the organization of employees who were or are 
enrolled in the Basic Skills Program. The original plan was to 
collect information on employee involvement in several ways, 
including: 

(1) an intake questionnaire for participants who are enrolling 
in the basic skills math class for the first time, to gather base- 
line data against which other data can be compared; 

(2) a "maintenance" questionnaire for participants who are 
enrolled in the program, to collect follow-up data in regularly 
scheduled intervals ~ perhaps every eight weeks; 

(3) an exit interview for participants who have completed the 
program; 

(4) an interview for supervisors; 

(5) an interview for participants who completed the program 
before the evaluation was begun to collect retrospective data. 

The intake and maintenance questionnaires ~ those parts of 
the evaluation system which have been developed - focus on 
whether a program participant has been asked to participate in or 
volunteered for a range of activities in the organization. These 
activities include: 
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• participation in teams/committees like Basic Skills, Material 
Waste, ISO, Safety, Total Quality Waste Team; 

• involvement in customer interface projects/initiatives like 
giving plant tours, surveying customers, visiting other plants 
and/or customer sites; 

• involvement in skills development projects/initiatives like 
college courses, GED preparation, job-related math skills, 
language classes; 

• involvement in training projects/initiatives like cross 
training, facilitator training; 

• involvement in special projects like community 
work/volunteer efforts (United Way), company-wide or team 
projects, special presentations; 

• involvement as a trainer or a tutor; 

• awards/recognition received. 

The intake questionnaire also includes a self-assessment of job 
attitude, job knowledge, production, quality of work, flexibility, 
and being a team player. 

Newer questions focus on involvement on the job as outlined 
above. More specifically: 

• There is now a co-worker assessment comparable to the self- 
assessment. It asks the participants to rate co-workers on the 
same criteria they have just evaluated themselves on. 

• There are five new broad questions adapted from the 
performance appraisal with branching questions for each area 
that assess the participant's awareness of problems in the area, 
the action needed to correct the situation, and the participant's 
record of taking some appropriate action. The questions are: 

~ Are there improvements which need to be made in your 
work area? 

~ How would you evaluate the morale in your work area? 
~ Do you think it is important to communicate between 
shifts? 

~ What is the last unsafe condition you are aware of? 

~ How is the work flow in your area affected by vacations 

or days out? 

The evaluation began with gathering data from the intake 
interview in September 1992. Team members understood that it 



17 225 



would take time to build up to the five-phase evaluation system 
they had plamied. However, they did not anticipate the problems 
that lay ahead, keeping up with data collection and mastering 
SPSS. 

They had planned that preliminary analysis of the early data 
would help to direct the next phase of the evaluation. They had 
anticipated that preliminary analysis of the data would shed some 
light on the answer to the question: does participation in the Basic 
Skills Program enhance employee involvement? They anticipated 
that long-term data collection and analysis would provide a more 
complete answer. 

The On-site Coordinator was responsible for conducting 
interviews with the majority of the program participants. He was 
assisted by members of the Basic Skills Committee. In the 
summer of 1993 the On-site Coordinator coded the data on all the 
questionnaires completed to date and thus prepared them for 
programming on the computer. The On-site Coordinator was 
inexperienced with computers, and eventually took a course to 
familiarize himself with the computer functions that would permit 
analysis of the data he had so carefully coded. 

A member of the support staff in the Training Department had 
been assigned the job of typing in the coded data. She had 
participated in the original SPSS training the previous February 
and was prepared to begin to work on SPSS. This person soon 
received a promotion and was replaced by someone who had to be 
introduced to SPSS. 

This new person received individualized instruction from the 
SPSS trainer. However, she injured her back, was out of work for 
a while, and by the time she returned needed additional 
instruction. She had not used what she learned quickly enough 
and needed retraining. After she was retrained she typed in the 
data but there were still problems. The data were not typed 
correctly. Other responsibilities made continued work with the 
data less of a priority for her. At this point, the Educational 
Coordinator took it upon herself to clean up the data and work 
with the trainers to achieve a preliminary analysis. 

The Educational Coordinator is a very busy person who had 
herself recently received a promotion. She was now responsible 
for overseeing the development of Basic Skills Programs in other 
Norton plants throughout the northeast region of the United 
States. She was motivated to master the elements of evaluation 
because she wanted to replicate the process in the new programs. 
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Unfortunately, she encountered a few problems when she tried 
to carry out her plan. The first problem was the volume of data 
she had to sort through. There were severa? iterations of 
questionnaires; hundreds of variables; and hi^ndreds of 
respondents. The second problem was that the data were more 
inaccurately recorded on the computer than anyone had 
anticipated. 

At this point the SPSS trainers might have taken over the 
clean-up and programming of the data. However, they continued 
to operate as outside resources. In retrospect, this was the result 
of how close to being cleaned up everyone thought the data were. 
These problems plagued the analysis until the last meeting. 

The facilitator had developed a plan with the Educational 
Coordinator and the SPSS trainers to spend two days at the Norton 
Company finally running some preliminary analyses. Again, the 
data were compromised and no real analysis was possible. 
Problems kept poppiag up that took hours to solve. 

These problems were all traced back to a short-cut that the On- 
site Coordinator had taken with the original coding of the data. He 
had coded the data on the questionnaires themselves instead of on 
proper coding sheets. This was time saving in the short nm and 
disastrous in the long run. (The SPSS trainer had cautioned 
against this kind of short cut but, in the end, gave the O.K.) It 
meant that there was no record of the placement of the coded 
items other than in the computer where they were clearly 
compromised. 

As indicated above, there is a plan to complete some analyses 
of the data. The Educational Coordinator will complete this work 
in her own time, culling what is useful for replication purposes. 



Analyzing the data 

The Process Evaluation 

These are the seventeen components of a quality program 
which the team reviewed, and the actions team members needed 
to take to bring the component into "quality range." If no action 
was needed, then "No action" is stated. 

Component #1: Articulate Philosophy of Program. 
No action 
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Component #2: Develop and Implement Outreach and 

Recruitment Plan. 

Action: 

• Recruit employees with lower skills 

- Tie recruitment to personal needs as well as company 
needs 

• Involve supervisors in recruitment 

Component #3: Provide Orientation 

No action 

Component #4: Establish Intake Procedures 
Action 

• Establish an intake procedure that defines a "career path" 
for the participant. 

Component #5: Design and Implement Initial 

Assessment Activities 

Action: 

• Think through a way of assessing if learning objectives at 
both the individual level and level of the class are being . 
met, and if the skiUs being transferred. 

Component #6: Develop Curriculum 
Actions: 

• Review the curriculum so that it supports the goal to 
enhance involvement. 

• Condense the curriculum and make it more Norton- 
specific. 

• Ensure that the instructional method supports the goal. 

• Continue training for instructors. 

Component #7: Select Appropriate Learning 
Arrangements 

No Action 

Component #8: Develop On-going Assessment 

Procedures 

No action 

Component #9: Ensure Support Services 
Actions: 
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• Forge linkages with higher education programs. For 
example, the GED program at Qjiinsigamond and college- 
level courses there. 

• Think of the program as a continuimi from basic skills 
support to intermediary support to college level survival 
skills, all in preparation for bridging to higher education. 
Joe Passeri and Kathy Rentsch can nm informational 
meetings at Norton; others can offer support/instruction in 
creative thinking and problem solving. 

Component #10: Staffing 
No Action 

Component #11: Staff Training and Development 
Action: 

• Continue training for instructors 

Component #12: Program Planning, Evaluation, 
Monitoring and Management 
Actions: * 

• Joe Passeri and Bethany Brockmeyer should meet at least 
once every two weeks to plan the facihtation of the Basic 
Skills Committee meeting 

• SoUcit supervisors for the Basic Skills Committee 

• Add instructors to the Basic Skills Committee 

• Make the meeting time of tiie Basic Skills Committee more 
convenient (For example, once a month make it earlier) 

Component #13: Tracking and Reporting 

Action: 

• Reduce the time it takes to process recommendations to 
the Steering Committee. 

Component #14: Fiscal Management 

Component #15: Facilities 

Component #16: Follow-up/Exit 
Action: 

• Docimient why people leave the program and use tiie 
information to enhance recruitment. 

Component #17: The Business Education Partnership 
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No action 



The results of actions taken from the process evaluation 
include: 

• Supervisors have become more involved in the Basic Skills 
Committee 

• The Basic Skills Committee has become more employee-run. 
Employees are making more suggestions about how to bring 
attention to the need for and benefits of the Basic Skills 
Program. For example, at a Basic Skills Committee meeting one 
employee suggested that the CEO for Bonded Abrasives be 
invited to comment on how education has value for 
modernization. He suggested that the comment become a 
paragraph in a subsequent company newsletter and that 
student quotes should also be added to this newsletter review. 

The survey 

To date, the team has been able to produce only the most 
preliminary analysis of the survey data. As described above, 
preparing for analysis has been a far more arduous task tlian the 
team members, facilitator or trainers foresaw. In summary, there 
are several reasons for this. 

• The coding process was long and arduous. There are three 
versions of a survey that has many variables. Each survey 
required its own coding scheme. 

• Changes in personnel in the training office required 
instructing a new person in coding and programming midway 
through the activities. 

• The SPSS consultants live several hundred miles away from 
Worcester. Although we did not anticipate the distance being a 
problem, the distance made it impossible for the consultants to 
provide the hands-on support that some Norton employees 
required while they mastered this data-base. 

• Changes in the structure of the company and in the positions 
of key personnel (other than support stafQ who supported the 
evaluation made it difficult to maintain continuity of focus on 
the project. 

• Changes in the structure of the company coupled with the 
amount of time that had elapsed since the evaluation began, 
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began to shift the perspectives of key players on the team 
about the most important indicators of success. 

However, the team remains mostly comfortable with the goal of 
enhancing employee involvement, especially on the job. The 
Educational Coordinator said: "It (still) measures a more realistic 
outcome than productivity or the quality of the product." The 
Manager of Human Resources is not sure that involvement in the 
organization (beyond involvement on the job) is still a meaningful 
outcome. 

One practical suggestion about how to speed up the analysis of 
the remaining data (and provide a more-simply digested set of 
results) is to aggregate categories in the survey into a single 
measure of involvement. For example, aggregate the categories 
"volunteered," "participated," and "tried" for Questions II A 
through H. "Involved" might be designated as a score of 1; "not 
involved" as a score of 0. (See Appendix ) 

Additional evaluation-related finding 

Five of the eighteen to twenty people who bid to be a group 
leader in May 1993 were from the Basic Skills Program. 



Preparing a strategy for reporting the findings 

The more-active members of the Basic Skills Committee have 
been informally reporting the results (in the broadest sense) of the 
evaluation to other members of the Basic Skills Committee and other 
Committees throughout the company as needed. Reporting in this 
case is not limited to a formal written report of results of the smvey. 
Such a thing does not yet exist and may not exist for some time. 
However, considerable discussion about the evaluation and its 
relationship to program planning has taken place and has moved the 
program forward. 



231 

23 



P hase IV : Deciding what happens next 



Taking follow-up action 

The team decided that: 

• The team has good basic information and it should be analyzed 
for its own sake. Start with an analysis of the demographics of 
the people who have participated in the program. Then proceed 
to a simple review of frequencies for all the variables for insights 
into what the program has achieved. 

• The team should analyze the current data before changing the 
instrument to see exactly what the current instrument is able to 
demonstrate. The objective itself ~ improved employee 
involvement - may have little appeal outside the training 
department, unless it can be linked to more bottom-line issues. 
The concerns of the other managers are more directiy linked to 
the bottom line. Is quality better? Are we producing more? Is 
diere more of a profit? The Basic Skills Committee is faced with 
the challenge of demonstrating to the management groups that 
this program is working in terms that make sense to them. 
Analysis of the current data may help to clarify how address this 
challenge. 

• If the evaluation focus on employee involvement is maintained, 
the size of the instrument should be reduced before being used in 
the future. There are too many data provided by the current 
instrument. While current data should be analyzed, future data 
can be reduced by modifying the instrument. Keep the common 
threads so the data will be comparable. 

• The team should make better use of existing information in the 
company and tie this information into evaluating the effect of the 
Basic Skills Program. One option is to focus on participant job 
performance and use the data available from performance 
appraisals to enhance the evaluation . 

• The team should involve the thirteen new businesses in Bonded 
Abrasives in a process of redefining the goals and objectives of 
the Basic Skills Program. This might mean establisliing thirteen 
Basic Skills Committees and a mechanism to correspond among 
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them. Opening up the process in this way will help the team to 
redefine the focus of the evaluation and to establish evaluation 
methods that can better assess if the program is helping 
participants do their jobs better and improve their lives. 

• Supervisors should be solicited to give concrete measures of 
improved employee performance on the job. This information 
should be merged with what is learned from a review of the 
performance appraisal. 

• The team needs to think through how to use evaluation results 
to improve the program. 

• The team needs to specify audiences for the evaluation results 
and what the team wants to say to them. 

Evaluating the evaluation 

These are comments which team members and the evaluation 
facilitator made about the evaluation process: 

• The Norton team should have considered hiring out certain 
aspects of the evaluation to consultants. Conducting a basic skills 
program evaluation of this scope is a big job. There comes a point 
when internal staff cannot extend themselves beyond their 
regular jobs and consultants should be used. 

• More hands-on technical assistance from the SPSS consultants 
would have helped the team. The SPSS consultants could have set 
up the database, ensured that data entry was clean, and generally 
worked more closely with the team. 

• The work of the evaluation went on for too long, losing 
continuity of purpose. Some of the reasons for this were not 
controllable. Core team members changed their jobs and this 
changed their roles on the team. Support staff also changed and 
required additional training and "catch up" time. But the time- 
line could have been tightened, especially by reducing the scope 
of the preliminary analyses. An earlier preliminary analysis 
might have inspired the team to go forward with the analyses in 
other manageable increments. 
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• The team agreed to continue to discuss the appropriateness of 
employee involvement as the central focus of the Basic Skills 
Program evaluation. The evaluation process has been valuable 
but the Manager of Human Resources is left feeling that the 
original instrument -- edited to include the new focus on 
employee involvement on the job -- is not measuring what the 
company wants to know about in a valid and reliable way. The 
Norton team redefined its evaluation focus midway through the 
project. In extending the focus to employee involvement on the 
job, the team came closer to evaluating "the competencies which 
are at the heart of what is needed to take the company where it 
wants to go." But improved job performance might be a more 
appropriate indicator. Discussion of these concerns should 
continue. 

• The Educational Coordinator is pleased with what she has 
learned and is poised to think tiirough how she can apply this to 
workplace education programs which she oversees in other plants. 

IV. REFLECTIONS 

Facilitator Laura Sperazi's comments: 

• The Norton team would have fared better if they had collected 
fewer data on fewer people - and mastered some of the 
fundamentals of data management ~ before trying to manage 
several data sets with hundreds of variables in each. The general 
lesson to teams which is derived from the Norton Company 
experience is: collect information as soon as you can, then analyse 
it promptiy; do not wait for the data to be perfect before you try 
to make sense of what you have. This is especially true for 
inexperienced teams who are just learning how to evaluate. 

• SPSS is a good database program but other, more simple 
programs would be less intunidatijig for teams to use. Lotus or 
Data Ease are good examples of these. If team members are 
familial- with Lotus or other database programs they should begin 
by using these familiar programs for evaluation purposes and 
then graduate to more sophisticated programs, if needed. 

• Everyone -- team members, faciUtators, consultants -- need to 
learn to be comfortable with the slow evolution of focus for an 
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evaluation of a workplace basic skills program. Knowledge grows 
along with time, staff changes, incremental opportunity to discuss 
items in new and different ways, and with lessons learned in 
other areas of work. 
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INTAKE INTERVIEW PROCEDURE 

Explain to the participant the purpose of the Interview: 

~ To determine the level of employee involvement^ both within 
Norton and outside the company, at the time the employee 
begins training in Basic Skills. 

- To track the change in employee involvement during the 
employee; s continued career at Norton Company (6 month 
intervals) . 

The first page of the Interview is separated from pages 2 - 
6 to keep information collected separated from the 
participant's name. Pages 2-6 are then identified 
numerically to maintain confidentiality. 

Information regarding race, gender, and age requested on 
page ?? is provided on a voluntary basis and only used in 
the department; be certain to explain this to the participant. 

Instruct the participant (s) to only fill out page one by 
themselves; the remainder is to be done as follows: 

- the interviewer reads the questions to the participants as 
he/she follows along on their own. 

- the interviewer records the participant's responses as they 
are given. 

Distribute Part VII - Introduction to the Training Department - 
and review with participant. 

Thank the participant for his/her help in collecting the 
required information. Invite him/her to contact the Training 
Department at ext. 2441 or 2145 for any additional training 
which might be of benefit, or for help in answering any 
questions regarding educational assistance, further education 
at colleges or technical schools, and career guidance. 
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X. EMPLOYEE INFOEMATION 



NAME DATE 

SHIFT (circle) 1st 2nd 3rd ROTATING 

LENGTH OF TIME AT NORTON 



WORK HISTORY Cl±st the jobs you hare held at Norton; start vlth your 
current job first) : 

DEPT. / DATE DATE 

JOB PLANT SUPERVISOR STARTED ENDED 

1 . PRESENT 



3. 
4. 



LIST OPERATIONS YOU ARE CERTIFIED ON ACCORDING TO ISO 9002 STANDARDS. 



TELEPHONE: HOME ( ) WORK 



LAST SCHOOL HIGHEST GRADE 

ATTENDED COMPLETED 



PROGRAMS/COURSES OUTSIDE OF NORTON CO. 
CURRENTLY ENROLLED 



LANGUAGE SPOKEN AT HOME 
OTHER LANGUAGES SPOKEN _ 

2/1/93 
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II, EMPLOYEE INVOLVEMENT PROFILE 



2 



For the following gestions (A. thru G. ; , please mark a V_ 
beside activities you volunteered for, an A beside those you 
you were asked to participate in, and a T Feside an activity 
you tried to participate in but for reasons beyond your control 
could not. Please indicate the year(s) of your involvement. 

A. Have you been involved on any committees or teams? 

MARK CCMMITTES OR 

y,A OR 37 YEARCS) TEAM 

BASIC SKILLS 



MATERIAL WASTE 



ISO 



SAFETY 



TOTAL QUALITY 
T.Q. W. 



TRAINING TEAM 



COMMITTED TO ACTION 



OTHER TEAMS: 



B. What customer activities have you been involved in? 

MARK CUSTOMER 
V,A OR T YEAR (S) ACTIVITY 



GIVING PLANT TOURS 

VISITING OTHER PR^S 
AND/OR CUSTOMER SITES 

OTHER: 



C. What educational programs have you been involved in while at 
Norton Company? 

MARK EDUCATIONAL 

V,A OR T YEAR(S) PROGRAM 

COLLEGE COURSES 

GED PREPARATION 

BASIC SKILLS CLASSES 

CERTIFICATE COURSES 

OTHER: 
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D. What Norton training courses have you been involved in? 
MASK TRAINING 
V,A OR T YEAR(S) PROGRAM 

LEADERSHIP COURSES 

FACILITATOR TRAINING 

TRAINING VIDEOS 

OTHER: 



E. What Norton community service projects have you been involved 
in? 

MARK SERVICE 
V,A OR T YEAR(S) PROJECT 

UNITED WAY (circle) 

GIVER, CAPTAIN, VOLUNTEER 

BLOOD DONOR 

TUG OF WAR TEAM 

OTHER: 



F. What Outside Community Service have you been involved in while 
at Norton Company? 

MARK CCMMUUITY SPECIFY 

V,A OR T YEAR(S) SERVICE DUTIES 

CHURCH 

SPORTS 

CHARITIES 

SOCIAL 

YOUTH ORG. 

OTHER: 
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Have you been involved in any of the following training 
activities while at Norton Company? 



MARK 
V,A OR T 



YEAR(S) 



ACTIVITY 

TUTOR 

CROSS-TRAINER 

SAFETY TRAINER 

NEW EMPLOYEE 
ORIENTATION TRAINER 

CLUSTER PRESENTER 

TEACHER'S AIDE IN 
BASIC SKILLS 

OTHER 



H. What special recognitions have you received while at Norton? 
MARK 



V,A OR T 



YEAR(S) 



JRECOGNITION 

RECOGNITION LETTER 

RECOGNITION BREAKFAST 

RECOGNITION LUNCH 

TOTAL QUALITY WASTE 
TEAM RECOGNITION 

CERTIFICATES 

PIONEER AWARD 

CHAIRMAN'S AWARD 

OTHER: 
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I. The next four questions have to do with your continued 
leamingr at Norton Company. 

1. Do you feel encouraged or supported to become more involved 
in the company and, if sOr how? 



2. Are there barriers which prevent you from becoming more 
involved in the company and, if sOr what are they? 



3. What might help you become more involved in the company? 



4.. Are there specific areas in which you would like to be more 
involved in the company? 



III. STUDENT'S OBJECTIVES/GOALS 

A. What are your educational goals? 



B. What are your career goals at Norton Company? 
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XV, SELF-ASSESSMENT DESCRIPTION 

Please rate yourself on the six performance qualities 
below, (circle one) 



JOB 

ATTITUDE: 
JOB 

KNOWLEDGE : 

PRODUCTION 
ON THE JOB: 

QUALITY OF 
YOUR WORK: 

FLEXIBILITY : 

TEAM PLAYER: 



EXCELLENT 

EXCELLENT 

EXCELLENT 

EXCELLENT 
EXCELLENT 
EXCELLENT 



VERY GOOD 

VERY GOOD 

VERY GOOD 

VERY GOOD 
VERY GOOD 
VERY GOOD 



GOOD 

GOOD 

GOOD 

GOOD 
GOOD 
GOOD 



FAIR 

FAIR 

FAIR 

FAIR 
FAIR 
FAIR 



POOR 

POOR 

POOR 

POOR 
POOR 
POOR 



V. CO-WORKER ASSESSMENT 

Please rate your co-workers on the same six qualities. 



JOB 

ATTITUDE : EXCELLENT 
JOB 

KNOWLEDGE : EXCELLENT 
PRODUCTION 

ON THE JOB: EXCELLENT 
QUALITY OF 

YOUR WORK: EXCELLENT 
FLEXIBILITY: EXCELLENT 
TEAM PLAYER: EXCELLENT 



VERY GOOD GOOD 
VERY GOOD GOOD 
VERY GOOD GOOD 



VERY GOOD 
VERY GOOD 
VERY GOOD 



GOOD 
GOOD 
GOOD 



FAIR POOR 

FAIR POOR 

FAIR POOR 

FAIR POOR 

FAIR POOR 

FAIR POOR 



THANK YOU FOR YOUR ASSISTANCE! 
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1) 



2) 



VX EMPLOYEE JOB INVOLVEMENT 



Are there improvements which need to be made in your work area? 



A. What improvements ought to be made to improve the work 



B. What efforts have made to help improve the situation? 



How would you evaluate the morale in your work area? 



A. What could be done to improve morale? 



' B. In what way could you improve moorale? 



C. How have you tried to improve morale? 
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communicate between? 3 Wv'^'^S 



3) Do you think it's important to 

YES - continue to A NO - continue to question 4 

A. Is it important to communicate information to the next 
^ J. person on your job? 



B. What is the best means to communicate information to the 
next person on your job? 



C. How have you established communication with the next 
person on your job? 



4) What was the last unsafe condition you are aware of? 
NO - go on to question 5 

A. How could the situation be made safer? 

B. What have you done to make the situation safer? 



5) How is the work flow in your area affected by vacations or days 
out? 

NOT EFFECTED - End of int'arview 

A. How could the work flow be better maintained during 
vacation time? 



B. What have you done to improve the work flow during 
vacations? 
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CRITERIA BELOW REFERS TO PART VI QUESTIONS. 

Listed below are the content areas questions 1-5 are designed to 
focus on. The Criteria refers to the Performance Appraisal areas 
that are also evaluated in the question. 

QUESTION #1 - IMPROVEMENT IN YOUR WORK AREA 

CRITERIA: QUALITY, CONTINUOUS IMPROVEMENT, AND INDIVIDUAL 
PRODUCTIVITY 

QUESTION # 2 - MORALE IN YOUR WORK AREA 

CRITERIA: CONTINUOUS IMPROVEMENT, RELATIONS WITH OTHERS 

QUESTION #3 - COMMUNICATION BETWEEN SHIFTS 

CRITERIA: CONTINUOUS IMPROVEMENT, PRODUCTIVITY, RECORD 
KEEPING, PJILATION WITH OTHERS. 

QUESTION # 4 - SAFETY 
CRI TERIA : SAFETY 

QUESTION # 5 - WORK FLOW « 

CRITERIA: FLEXIBILITY, SKILL, CONTINUOUS IMPROVEMENT, 

PRODUCTIVITY 
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Joe Javorski 

Norton Company 

Bonded Abrasives Division 

1 New Bond Street 

Box # 15008 

Worcester, MA 01615-0008 



F-203 University Park 
Uptown Road 
Ithaca, NY 14850 
January 18, 1993 



Dear Mr. Javorski, 

Laura Sperazi, of Evaluation Research, recently sent us a 
copy of your survey instrument for gathering maintenance data on 
your basic skills programs. She mentioned that you would 
appreciate some feedback from us before you actually start the 
data collection. We hope you find the following of some use. 

We were not quite sure of the precise questions which you 
hope to address with this instrument, although the 'Introduction 
and 'Nature of Program' sections helped set a context. Will the 
workers see these two sections before responding? Including this 
section may bias their answers, since it suggests a set of 
'socially desirable' responses. Also, we were unsure how the 
instrument would be administered, so some of our comments may be 
moot points. 

In general, we liked the instrument's layout. However, we 
do have some comments. One of the issues researchers often must 
address is the confidentiality of the responses. You may want to 
include a separate page for the respondent's name and phone 
number, and indicate in a prefatory paragraph that this 
information will be separated from the rest of the survey prior 
to the coding and analysis stages. You could then put a number 
on each interview (on each page, in case they get separated) so 
you could keep tabs on who gave which answers, should this be 
necessary. While the respondents may realize that the 
information is still traceable, they should f6el that there is 
less connection between their responses and any possible 
evaluation of them as workers. This becomes more important with 
the introduction of a control group, who most likely will not 
know the purpose of the study. The prefatory statement could go 
on a cover page, which would also state who is doing the study 
(i.e., is it being done under the auspices of Norton, the hxuaan 
resources department, is there any union involvement, or is this 
being done under the auspices of an outside group?). 

(I) In the first part on 'Employee Information', we suggest 
a few stylistic changes. (1) Under work history, is the 'Most 
Recent Job' intended to be the current job or first previous job? 
If it is current, then the two previous questions on plant/dept 
and current supervisor's name are redundant. If it is the 
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previous job^ then you do not appear to ask what the current job 
is, though this information may be useful. This would also 
ensure that everyone answers in the same manner — you want to 
avoid having some people list current job in the first slot while 
others put previous job in that slot. 

(2) What is the line under plant/dept for? Presumably^ it 
is a wrap-around from the previous line^ but it is distracting. 
The responses (1st, 2nd, 3rd) need not be underlined. 

(3) For the shift question, we would line up the word 
'shift' with 'circle' on the left, and put the word 'circle' in 
parentheses and in small letters. 

(4) Why is there a question mark at the end of the statement 
telling the respondent to list operations on which they are 
certified? We assume they understand what ISO 9002 standards 
are. 

(5) The languages spoken question may lead to confusion: it 
could be interpreted either as asking for only those foreign 
languages spoken (as an English speaker might) or for all (as a 
foreign language speaker might). It would then be unclear 
whether a response of, for example, 'French' means that the 
person speaks English and French or that the person speaks French 
but does not yet feel qualified in English as a second language. 
This could be avoided by pointing out to include (or exclude) 
English. This is somewhat a moot point if they are reading the 
survey rather than having an interpreter read it to them, because 
English must be in their repertoire of languages — as we do not 
know the details of the administration of the instrument, we 
included this comment. 

(6) It may prove useful to gather information on the 
respondent's gender and race as well, although this is sensitive 
information. This would allow you to make distinctions between 
categories of people. For example, ic may turn out that the 
programs create different outcomes and reactions among these 
groups, such as between men and women. Finding this out could 
then lead to appropriate changes in the programs themselves, in 
order to increase their utility. There may also be differences 
between groups in the forms of employee involvement. 

While some of the comments may seem picky, respondents may 
get briefly confused or annoyed at this early stage, and become 
less willing to invest time in doing you a favor and helping you 
on your study. Making them as comfortable as possible pays off 
in the end results, and hence researchers have ended up paying a 
lot of attention to minute details. 

(II) In part II, the 'Employee Involvement Profile', we 
liked the general approach. (1) The placement of the word 'year' 
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may present problems. Since it is off to the side, and there are 
no slots for the answer, it may be missed by the respondent. As 
well, it may be useful to change it to "YEAR(S)", so that people 
do not feel restricted to one year. It might be useful to have 
two slots in. front of each category, one for the 'volunteered' 
versus 'asked' distinction, and one for the years of involvement. 

(2) Are you interested in having a category for activities 
which a person tried to join but was excluded for whatever 
reasons (e.g., space limitations, qualifying criteria)? This 
question may prove to be more trouble than it is worth, since it 
may evoke socially desirable answers, but could be of some 
benefit conceptually, and may be worth trying. 

(3) In sections B and E, in the 'Other' response, there 
should be a line for the respondent to explain what they did. 
There are several reasons for this. One is to ensure similarity 
of layout over sections. Second, thjese are both very likely to 
evoke a 'yes' answer since this would be a socially desirable 
response. By having them describe the activity, they cannot just 
answer yes in some vague way. This information may prove useful 
in creating a new code for the analysis after the data have been 
gathered . 

(4) In sections C, G, and H, do these questions refer to any 
point in the respondent's lifetime, or only while at work at 
Norton? Some respondents may answer one way and others in the 
opposite manner. 

(5) In section E, will the respondents know the difference 
between the types of United Way activities? It may be useful to 
have one line for United Way, with the three types of involvement 
listed afterwards with an indication to circle the appropriate 
activity. 

(6) In sections C and F, there should be an 'Other' slot, in 
case the respondent feels uncomfortable making use of your 
predefined categories. 

(7) In section F, it may be useful to offer more lines, in 
case they are quite involved, 

(8) Is it possible that a person could be asked to be 
involved in a program one year, and volunteer for the same 
program in another year? You may want to anticipate how you 
would code a response indicating both a V and an A (e.g., does 
one take conceptual priority over the other). 

(9a) In section I, there seem to be several leading 
questions. In the prefatory comment, the statement that Norton 
is hoping for employees to become involved will put pressure on 
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the respondents to find some form of involvement, even if this is 
not their true feelings. We would remove this sentence. 

(9b) Question 1 should read, "Do you feel encouraged or 
supported to become more involved in the company, and if so, 
how?" Phrasing the question in this form will allow people who 
do not feel encouraged to say so. In the current form of the 
question, you will not be able to tell whether a blank response 
means the respondent has negative feelings about support or has 
not been able to identify mechanisms of support. 

(9c) Similarly, question 2 should read, "Are there barriers 
which prevent you from becoming more involved, and if so, what 
are they?" If you are concerned that this may be perceived by a 
respondent as a risky question to answer (they may feel that 
answers which criticize the company or management may lead to 
some form of retribution against them) , you can leave it in its 
present form. This way, people may have a bit more freedom to 
make critical comments. 

(9d) A fourth question might be of use: something along the 
lines of, "Are there specific areas in which you would like to be 
more involved in the business?" This would be particularly 
useful with the addition of a control group. 

(III) Part III seems fine. However, coding this for 
analysis may prove to be a bit of work (although do-able). 

(IV) Part IV has a nice layout, although there is a lot of 
blank space staring at the respondent. Beyond filling in space, 
it might be useful conceptually to include a Part V, which has a 
layout identical to Part IV. This section would ask the 
respondent to rate (on average) their coworkers on the same 
criteria as in Part IV. This would give you the ability to see 
how workers view the general culture of the firm, as well as to 
ground the self-evaluations of the respondents against their 
perceived context (for example, a male respondent who views 
himself as having a good Job Attitude while perceiving his 
coworkers as having excellent Job Attitudes is a very different 
creature from a respondent who views himself as having a good Job 
Attitude while perceiving his coworkers as having poor Job 
Attitudes) . 

(V) Some additional comments. (1) Will the final version of 
the survey have the "sdp" in the lower left corner of the final 
page? Does this serve a purpose? 

(2) A section of open-ended questions (similar in 
construction to Part III) asking the respondent to evaluate any 
benefits and/or drawbacks they received from involvement m the 
programs listed in Part II might be of some use to your study. 

(3) Finally, it may be a nice touch (and one which 
respondents really do appreciate) to have a final paragraph 
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thanking the respondent for taking the time to share their 
thoughts . 

(4) Let us reiterate that we thought the survey useful and 
well thought out. The questions should lead to some interesting 
analyses in their own right, as well as in comparison with the 
baseline data. 

(VI) Laura Sperazi also mentioned that you had some concerns 
about the use of a control group. There are two reasons for 
gathering infontiation from such a group. The first is to 
eliminate any potential biases which might result from broad 
changes which occur between the time of administering a program 
and the gathering of follow-up data (such as a recession which 
could lead to job security concerns). If the control group 
responses change in a similar manner as the experimental group, 
then the change is probably not due to the program itself, but 
rather some broader event. If the change is evident in only the 
experimental group, then the program seems to be the source of 
these changes in response. This use of a control group requires 
data be gathered from non-experimental respondents at both steps 
of the data gathering. Therefore, this use may therefore not 
apply in your study. 

The second use is to compare the responses of the post- 
experimental respondents to the general set of attitudes in the 
population from which they may be drawn. If the purpose of a 
program is to bring people 'up to speed' , then how close that 
goal has been achieved can be gauged by comparing the program 
graduates with other workers. If the purpose is to change the 
general attitudes, then the divergence between the program 
graduates and other workers would be of interest. Either way, 
this use would require gathering data on a control group as well 
as the experimental group. 

Gathering data on a control group is fairly easy. The 
instrument should be identical to that given to the experimental 
group in order to avoid any potential biases resulting from the 
use of different surveys. Similarly, the survey should be 
administered in an identical manner. The ideal method for 
identifying respondents for the control group would be random 
selection. Such a selection technique could result by taking an 
alphabetized list of all workers of the appropriate ranks subject 
to the programs (for example, if management is not eligible for 
the programs, then they should be excluded from the master list). 
You would then select every nth person — for example, every 20th 
person. The selection number would be such that the control 
group would be roughly the same size as the experimental group. 
The list should include those people who have gone through the 
programs, but they would be dropped and the next person on the 
list could be chosen. Also, if there are any other restrictions 
on entering the programs (e.g., by minimxim or maximum age, 
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gender, or race), the list should also reflect these selection 
criteria. 

Another method of selection, which is much more time 
consuming, is to match each experimental group member with a 
control member. This involves selecting the control person who 
is of the same age, race, gender, has worked for about the same 
amount of time in the company, in the same department, with a 
similar background (i.e., family socioeconomic status, 
educational attainment level ) • 

The purpose of either of these methods of selection is to 
prevent the selection method itself from influencing the results 
(for example, if a single department was chosen as a control 
group, then elements such as departmental relations could 
influence the results and create an apparent difference between 
the control and experimental groups). 

We hope that this feedback proves to be of some help. Best 
of luck, and we look forward to meeting you. Please feel free to 
call us at (607)257-7566 if there are any questions. 



Sincerely, 



Steven Andrews & Alice Ober field 



cc: Laura Sperazi 
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EVALUATION 
CRITERIA 



|sAFETY 

The digrtt to which the 
individual works in a sate 
vnanner, using proper 
ncechntQutSt tQuioment and 

protection. 



NEEDS 
IMPROVEMENT 



6000 
PERFORMANCE 



EXCEPTIONAL 
PERFORMANCE 



Does not always use 
aopropriatt caution and 
procedures. Is injured as 
a result of commtttinQ an 
unsafe eett or observed 
comrnirtftg unsafe aets> 



lUAUTY 
The degree to which the 
pmoioye^'s work is neat« 
f rderfy. free of error and 
satisfies the ctmomer 
^internal or external)* 



The work is sometinies 
less than acceotable to 
the internal or external 
custonr^er. Improvement 
is needed. Pride in 
workmanship is lacking^ 



Works in a safe manner 
using prooer techniques, 
equipment and 
protection* 



Work Is consistentiy 
neat, orderly and free of 
error Output satisfies 
internal and external 
customer. Rate of rework 
and rejections meet 
department goais* 



SKILL 

le ability to aopiy one's 
lowledge readily and 
effectively in the 
jjj^xecution of work. 

JNDIVIDUAL PRODUCTIVITY 
ne amount of quality work 
:omDieted or produced in a 
timely manner* Based on 
leasures and observations* 



Often needs assistance 
with job functions. 
LImite<) in potential work 
assignments. 



Can do the fuU range of 
the primary job 
requirements witii 
positive results. Needs 
tittle or no assistance* 



Suggests improvements to 
procedures or conditions to 
improve safety. Actively 
participates in safety 
initiatiyes. 



Work is consistentiy neat, 
orderly and free of en^or. 
Output satisfies intemal and 
external customer. Rate of 
rework and rejections 
consistently exceed 
department goals* 



Ability beyond what is 
required in the job 
responsibilities. Utiiizes \ ^ 
understanding of the oroctssX 
to anticipate problem areas. \ 



:oNnNuous 

JMPROVEMEPTT 
le degree to which the 
smployee contributes to 
jrocess improvements. 



Sometimes doesn't 
display full effort. Balow 
average results vs. team 
expectations. 



Not open to changes or 
process improvements. 
Designed to increase 
quaihy, efficiency and/or 
reduce cost. 



Displays strong effort on 
a regular basis. Regularly 
performs at a level 
consistent with team 
expectations. 



Makes suggestions for 
improvement within 
his/her area of 
responsibility. 
Implements procedures 
that increase quality, 
efftdency and/or reduce 
cost. 



Proactive * takes steos to 
increase produc^vity • 
methods, procus changes, 
combining operations, greater 
flexibility, etc Displays extra 
effort and consistentiy 
outperforms team 
expectations. 




Actively makes suggestions \ 
to improve the total process, \ 
increase quality, efficiency 1 
and/or reduce cost. Looks 
beyond own responsibilities. 



FLEXIBILTTY 
ie degree to which the 
hTployee willingly accepts 
job interruptions - temporary 
assignments in another area 
.and uses additional skills to 
Kpedite job progress. 



eric" 



Is somewhat rigid - not 
open to accept job 
Interruptions or 
assignment changes. 
UnvrilUno to work 
overtime or change 
scheduled shifu if 
needed. 



Winingiy accepts job 
intenruptions and 
changes to nonnal 
schedule or routine to 
Increase customer 
service or efficiency. 
Willing to work overtime 
and change shift 
schedule. 



Uses additional skills to^^^D l 
expedite job progressJ^TIiiinij 
and able to perform most of 
the essential operations in 
the area. Very open to 
schedule changes, eg. 
overtime, shifts. 
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EVALUATION 
CRITERIA 



RELATIONS WITH OTHERS 
'The degrftft to which tht 
•mploytt cooperatts, 
communicatttr and works 
.effectively with others*- 
treats others with 
dignity and respect 



TRAINING 

The degree to which thiii 
individual I cams new skiils - 
acct flts training 
opportuniijes. Willing to 
teach/share skills vtrtth ^ 

^^^^^^^ 



NEEDS 
IMPROVEMENT 



Frequently has dtfficuity 
communicattng or 
working effectively with 
others. Avoids resolving 
conflicts with others* 



Resists learning new 
skiils, unwilling to accept 
training opportunities. 
Has difficulty training 
others. 



GOOD 
PERr<)RMANCE 



Communicates and 
works effectively with 
others. Respects the 
feelings of others* 



Open to learning new 
skills and accepts 
training opportunities. 
Wining to teach/share 
skills with others. 



EXCEPTIONAL cL 
PERFORMANCE 



Works exceptionally well | 
with others. Proactive in ^ 8 
resolving issues directiy with v ^ 
supervision and eo-workers. \ ■ 



Strong motivation and desirej^ 
to help others learn and/or^ 
seeks additional training 
internally or externally to 
improve personal 
effectiveness. 



MAINTENANCE OF TOOLS, 
EQUIPMENT AND WORK 
AREA (HOUSEKEEPINQ) 
The degree to which the 
employee exercises care in 
the use, maintenance, 
security and availability 
(prompt return) of company 
tools and equipment* 
Maintains housekeeping. 



Work area is not left 
consistently clean and 
orderly. Tools or . 
equipment are misused, 
not used as intended, or 
not returned promptly. 



Consistently keeps a 
clean, orderly work area. 
Appropriate use of tools 
and equipment, including 
prompt return. Reports 
maintenance needs* 



Anticipates and reports 
maintenance needs and 
follows up. Looks beyonT^ 
own work area If needed to 1 
identify and address 
maintenance issues* . ^ 



RECORD KEEPING 
The degree to which the 
employee completes and 
maintains timely, neat, 
legible and error^free 
records. 



Sometimes submits 
incomplete, inaccurate, 
illegible or untimely 
records. 



Records are consistentiy 

maintained In a timely 

fashion. 

•Neat 

Organized 

•Legible 

•Accurate 

•Complete 



Suggests improvements, ] 
helps others. Takes extra 
steps to increase usefulness 
of records. Identifies trends- 
uses data. 



TARDINESS 

The degree to which the 
employee is on time for worl^ 
and remains at work for the, 
full shift 



A 



^even or more 
occurrences of tardiness 
and/or leaving early 
without prior approval. 

# Of Occurrences 



Four « sbc occurrences of 
tardiness and/or leaving 
early without prior 
approval.' 

# Of Occurrences 



Three or less occurrences of 
tardiness and/or leaving early 
without prior approval. 



# Of Occurrences 



ATTENDANCE 

The degree to which the ^ ^ 
employee is present for 
work. Except vacation, long 
service day, military duty, 
mf^fsi^ >iavet jury duty. 



g|^(^)Sionate leeve, 



duty* 



1^ leave and witness 



Six or more occurrences 
of absence. 



Four * five occurrencea 
of absence. 



Three or less occurrences of 
absence. 



^Of Occurrences 
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D€V€LOPM€NT flSSISTflNC€ CORPORfiTION 



410 Dover Point Rood Dover, NH 03820 (603)742^300 FRX (603) 749-0331 
MEMO 

TO: Laura Sperazi 



DATE: April 5, 1993 

SUBJECT: Feedback to Norton Company on Measures for "Employee 
Involvement On the Job" 

On 3/19, Bethany faxed a copy of a set of criteria (untitled; 
attached, with my handwritten comments) , with performance category 
definitions for each, and asked for my "feedback as to which would 
measure the involvement we are looking for." I called her on 3/22 
with the following feedback: 

1. All of their 12 criteria get at employee involvement on the 
job, not necessarily in the categories' descriptions 
themselves, but at least in the performance definitions given 
on a continuum in the three right-hand columns. Therefore, 
this looks like a very good basis for assessing their target 
behaviors . 

2 . Some of their criteria, and attendant performance 
descriptions, cover more than one phenomenon and might be more 
reliable assessment devices if broken into smaller bits. For 
example , the TRAINING criterion really includes three 
phenomena: a) learn new skills, b) accepts training 
opportunities, and c) willing to teach/share skills with 
others. I suggested they scrutinize each criterion and set of 
performance descriptions to look for other such multi- 
phenomena . 

3. I asked where this set of criteria came from and how they 
anticipated using it. She indicated that it was from their 
new internal review system, which might not be accessible to 
the Training Department's evaluation uses for a year or more. 
Thus, they may use it as a framework to develop their own 
rating system now. I suggested that, if they do so, they 
consider breaking each criterion down into more specific 
statements, such as many of those in the Aerovox form, for 
self- or supervisor-ratings. This would be more time- 
consximing to develop, but would yield more reliable results . 
The criteria shown on the attached pages, however, is really 
quite good, and it may not be necessary to develop it much 
further. It depends upon the extent of development resources 
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L. Sperazi 
Page 2 

they want to put into it, and how they'll use the results 
(i.e., "the precision of measurement needed depends upon the 
uses of the information"). 

I asked that she let me know what they end up doing with this, just 
to satisfy ray curiosity and bring some closure to the discussion 
for me. She indicated she would. 



cc: B. Brockmeyer 
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Mayl7,1993 



Don, 



As you will notice, the five questions devised have additional 
questions which are designed to ascertain differing levels of 
involvement . 

The first level is an employee's awareness of the situation the 
questiQn initially asks. 

The second level (question lA or 2A, etc.) is aimed at their 
awareness that something could be done to change the situation. 

The third level (ques. IB, 2B, etc.) asks if they have actually 
done anything or implemented an action to change a situation. 

On some questions it is noted that a "No" or negative response ends 
that particular question. 

The questions have a correlation to the Performance Evaluation so 
it could be used in relation with the Evaluation Questions. We 
also kept the questions non-job specific to allow for a more 
inclusive population given the variety of jobs we are drawing from. 

I hope this gives you enough information to go on. If not, please 
contact me at 508 -795 -2441. 

Thanks for your time. 
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D€V€LOPM€NT fiSSISTRNC€ CORPORATION 

410 Dover Point flood Dover, NH 03820 (603)742-6300 FflX (603) 749-0331 



MEMO 

TO: Laura Sperazi 
FROM: Don Cichon 
DATE: May 18, 1993 

SUBJECT: Feedback to Norton Company on 5/17 Quetitions for Assessing 
Employee Involvement on the Job 

Yesterday Bethany faxed me a set of questions they're considering 
adding to their Employee Involvement Profile, and asked for my 
reactions. I called her this morning and discussed the following: 

1 In general the questions looked good. There is a certain 
conceptual elegance in the three-level branching, with each of 
the three levels indicating important information to the 
company and facilitating a good thought flow for the 
respondent. There is also methodological soundness in their 
having derived the questions from the Performance Evaluation 
categories, since that will facilitate ties of this interview 
information to the Performance Evaluation system for validity 
and reliability purposes. 

2 My main concern is with scaling the responses to the 
questions. That is, comparability of answers will be 
difficult. For example, let's say morale is generally known 
to be poor in the packaging department because management has 
been considering reorganizing it, and employees therein are 
not sure of the status of their iobs for the future. In 
asking an employee question 2c), for instance. How have you 
tried ^o improve morale, one person may respond that he goes 
around trying to be cheery and encourage others to "keep their 
chins up." Another, however, may gather the group together at 
lunch every other day to discuss their options, try to 
separate rumors from facts, think out the real implications 
for various alternatives, pose questions to management, etc. 
It would seem that the tv;o employees' positive responses to 
this question should not be counted equally. 

As another example of comparability problems, if in a given 
department there is little change, in say safety involvement, 
from time 1 to time 2, does that mean that there's little room 
for safety improvement because it is already a very safe 
department or that the employees are unaware of or can't think 
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Page 2 



of anything to do about safety problems? You won't know just 
from tallying positive or negative responses to the questions. 

What the foregoing concerns point to is not to abandon the nev/ 
questions, but to be very careful in interpreting the 
responses in the context of the specific part of the plant and 
the conditions at the time they are asked. 

The other upshot of the concerns about scaling is that I think 
that any numerical scale developed will be inadequate to 
address those concerns, and so these nev; questions should be 
used more for the insights and patterns of "grounded" examples 
of how employees are involved, and not so much for numerical 
evidence of the impact of the training program on involvement. 
As long as these questions are supplemental tc the larger 
survey, then that is fine both purposes, a) "hard," 

statistical , summary evidence to management and b) realistic 
employee scenarios and insights, can be served. 



Question 
needs an 



'c" of every set 
outside source 



valid. Bethany indicated 
reviews for this purpose. 

Call me for further questions. 



asking about what you have done, 
f verification to be considered 
using the employee performance 
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TEAM EVALUATION AT PROTECT REACH : 



CASE STUDIES FROM TWO WORKPLACE EDUCATION 

PROGRAMS 



Prepared 
by 

Paul Jurmo 



Two of seven case studies in 
" Team Evaluation : 
Case Studies from Seven Workplace Education Programs," 
a report prepared in June 1994 
by Laura Sperazi and Paul Jurmo 
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PROTFCT REACH : 
CIVIL SERVICE EMPLOYEES ASSOCIATION 

AND 

THE NEW YORK STATE GOVERNOR'S OFFICE 
OF EMPLOYEE RELATIONS 



Introduction 



In 1986, tiie State of New York (through the Governor's Office 
of Employee Relations, GOER) and the Civil Service Employees 
Association, Inc. (CSEA) established a jointly-administered education 
program for state employees. Titled Project REACH, the program 
provides a wide range of basic skills educational services. 

This two-part case study examines how two REACH sites - one 
at the State University of New York (SUNY) at Stony Brook and the 
other at SUNY Albany - have adapted the team evaluation concept 
to their particular programs. These sites differ from the other cases 
presented in this report, because they were facilitated not by the 
authors of this research but by two adult educators trained by one of 
the authors and another specialist in collaborative planning. 

As such, their experience might be more representative of 
other programs in which people with limited prior experience with 
the team evaluation methodology are "out there on their own," trying 
to adapt it to their situation. For readability's sake, the two cases 
are presented separately, although in the early stages the two site 
facilitators did some preparatory work together. 
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SUNY at Stony Brook 



I. PROGRAM BACKGROUND 

In 1992, the assistant director of SUNY Stony Brook's Residence 
Hall Operations attended a REACH-sponsored seminar on the 
potential of workplace education programs. Management had 
concluded that approximately 25 percent of its workforce might 
benefit from a workplace basic skills program. In early 1993, REACH 
was asked to explore setting up some kind of program, presumably 
focusing on basic reading and writing skills. As it does at all its sites, 
REACH asked that SUNY set up an advisory committee which REACH 
staff could in turn discuss options with. 

REACH consultant Muriel Medina was contracted to conduct an 
organizational needs analysis (ONA), to clarify educational initiatives 
which REACH and SUNY might pursue. 



II. BACKGROUND ON THE TEAM 

History of the team 

The advisory committee set up at REACH'S request constituted 
a planning team which Muriel could work with to carry out the ONA. 
The resulting ONA report (whose executive summary Muriel 
distributed in both EngUsh and Spanish) identified math, reading and 
writing, and ESL as educational initiatives which the department 
might undertake. This range of sendees extended beyond the "basic 
reading and writing" focus which management originally had 
anticipated for the program. It in fact came as something of a 
surprise to management thc.t employees expressed so much interest 
in ESL, because they had had access to university-provided ESL 
courses for some time. However, th.ose courses were not tailored to 
the specific needs of those who attended, and thus apparently didn't 
capture the employees' interest. 

The committee decided to set up one class for the summer, a 
seven-week math course. It was agreed that Muriel would serve as 
coordinator, helping the team to set up and oversee this and possibly 
other courses. The committee felt that, by beginning with math, the 
program would avoid being seen as "just a program for the non- 
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English-Speakers and custodians." Math would be a "crossover" 
course, mixing people of different language backgrounds and 
departments (i.e., custodians and maintenance employees). After 
that course, a second course on "technical communications" (i.e., 
reading and writing) was implemented. 

After each course, Muriel faciUtated a team evaluation process 
described below. Both courses were deemed to be a success, and it 
was agreed that a third course - on ESL -- would be offered. 

Stakeholders represented on the team 

The advisory committee with whom Muriel worked was 
composed of three higher-level managers from the residence hall 
operations department, the union president, seven other custodial 
and maintenance employees, and two representatives from Project 
REACH (one representing GOER and the other from CSEA). 



III. THE PROCESS WHICH THE. TEAM WENT THROUGH 

Phase I : Facilitator prepares herself and prepares 
the team. 

Joint preparations with coordinator from SUNY 
Albany 

The "facilitators" of the team evaluation process at the two 
SUNY sites were the site coordinators themselves. This fact 
distinguished them from the other cases profiled in these case 
studies. In those other sites (Bull Worldwide Information Systems, 
Datatec Industries, Haartz Corporation, Norton Company, and Victoria 
General Hospital) the facilitator role was played by "outsiders." 
These outsiders had considerable experience with the team 
evaluation methodology and a significant investment in that 
approach. At the SUNY sites, the coordinators had an interest in the 
team notion, but no real prior experience using it. 

The two coordinators (Muriel Medina at Stony Brook and Carol 
Young at Albany) agreed that, to prepare themselves for their roles 
as evaluation team facilitators, they would get some extra up -front 
help. This help came in the form of on-site workshops and informal 
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telephone consultations provided by outside resource persons Paul 
Jurmo and Sue Folinsbee. 

These interactions between the two outside resource persons 
and the two site coordinators took the following forms: 

March 199 S workshop : On March 17, 1993, Sue FoUnsbee and 
Paul Jurmo conducted a workshop in Albany on the organizational 
needs analysis (ONA) methodology developed by Sue in Canadian 
workplaces. The workshop was hosted by Project REACH for the 
purpose of introducing staff of REACH-funded basic skills 
programs to the ONA methodology. ONA was presented as an 
initial step in a larger program planning cycle. ONA was a process 
in which stakeholders collectively examine what makes for a 
productive employee, how basic skills play a role (or don't) in 
productivity, and what initiatives a workplace might undertake to 
enable each employee to reach his/her potential. Participants 
analyzed how they might gather and analyze information in a 
simulated ONA. They emerged from the workshop with an 
understanding not only of the ONA methodology per se but of the 
notion of involving all stakeholders in an integrated planning 
process. 

Tulv 1993 workshop : By early July, Carol Young and Muriel 
Medina had each carried out ONAs in their respective sites. In the 
process, they had set in place a team planning process which they 
wanted to continue. Although it was still early in that process, 
they wanted to prepare themselves to lead their teams through an 
evaluation when the appropriate time arrived. 

To do so, Paul Jurmo met with them in Albany on July 6 th for a 
five-hour workshop. Carol and Muriel were introduced to the 
initial steps of the team evaluation process outlined below. They 
were challenged to anticipate how their respective teams might 
answer the questions of "who wants what information for what 
purpose?," what are the goals of the program?" and "how might 
data be gathered and from what source?" 

Muriel and Carol shared their responses to those questions, 
finding that they had much in common. In the process, Carol, 
Muriel, and Paul built a closer working relationship, agreeing to 
lend each other mutual support as they proceeded with their 
evaluation work. It was agreed that, now that Carol and Muriel 
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had thought through some of the mitial steps of planning a team 
evaluation, they would be better prepared to discuss evaluation 
issues with their teams in the coming months. 

Telephone feedback : In this preparatory period and 
subsequently, Carol, Muriel, and Paul had informal telephone 
discussions, giving each other updates and feedback as needed. 

By participating in these preparatory activities, the two site 
coordinators "did their homework" for the work which lay ahead. 

Facilitator prepares her evaluation team 

As stated above, Muriel was presented with an already- 
existing advisory committee when she was hired to plan and 
coordinate the Stony Brook program. As Muriel led the committee 
through the organizational needs analysis, members gradually began 
to create an identity for themselves as active decision makers rather 
than as a passive "rubber stamp" group merely giving formal 
approval to decisions made by others. 

As the planning process turned to the question of what role 
evaluation would play in the programs, Muriel introduced the 
advisory committee to this more-specific topic at a meeting on July 
21st. She discussed the continuing role of the committee and the role 
which evaluation might play now that the summer math course was 
underway. She used handouts to explain (1) the advantages of a 
team-based evaluation over traditional "external evaluator" models, 
and (2) the roles team members might play. 

Phase II : Planning the evaluation 

Clarifying program goals 

In the July 21st meeting, Muriel not only discussed the notion 
of team evaluation, she led the team through the step of clarifying 
program goals. In this case, the team brainstormed the full range of 
goals which might be served by REACH, rather than just the summer 
math course. These broad goals were identified as follows: 

Overall goal : to have training be interesting and useful. 
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Personal development 

• provide opportunity to go to promotion interviews. 

• improve self-esteem. 

• to make people hungry for more learning. 

Work atmosphere 

• meet the needs of Residential Operations. 

• improve morale even more. 

• create a feeling of caring. 

• have better employees. 

• make sure information gets to all employees. 

• move toward 50% release time and 50% worker's own time. 

Information/Skills 

• improve productivity. 

• offer refresher courses to Cleaners in Spanish as well as 
English. 

• improve English language skills. 

• improve job-related skills. 

• improve job-related reading skills. 

• provide opportunity to receive GED. 

• offer job-related courses. 

Program as a model for other workplaces * 

• expand, possibly, to other departments, allowing their 
employees to participate if there are openings. 

• share what we're doing at Residential Operations and how 
we've gone about it, if other departments want to model a 
program after ours. 

• This was rated as less important compared to other goal 
categories. 

Identifying who wants to know what information for 
what purpose 

In the July 21st meeting, Muriel displayed three flipcharts with 
the headings of "Who," "What information does the person want to 
know?" and "For what purposes? (Why does the person want to 
know?" Team members brainstormed the following responses: 
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Who 


What information 


For what purposes 


Supervisors 


• Did they learn 
anything in class? 

• Are they able to use 
wnat tiiev nave learneui 


• To make supervisor's 
job easier. 


Person attending class 


• What do they need 
improvement in? 

• Was best information 
given in class for their 

need? 


• To better themselves. 

better about 
themselves.) 

• For career 
advancement. 


Co-workers 


What information was 
learned in class? 


• To communicate better. 

• To improve teamwork. 


Management 


How to make 
information from the 
worKsnop av uiiaDiu lu 
other workers? 


• Better productivity. 

• Save money. 

• fmnrnvp splf-pstppm 


All employees in 
Residential Operations 


• What was learned in 
class? 

• What levels are the 
courses? 


• To better understand 
what the class was about. 

• To hplr^ \v/nrlv fnrpp 

•To know if they are 
qualified to take the 
courses 


Local-level union 


How is the program 
being offered? 


• Is money being spent 
wisely? (It will affect 
spending for future 
programs.) 


Union at other levels: 

• Regional director 

• State president 


Have I (local union 
president) done my job 
to see if program has 
been administered fairly 
to all? 


• Budgetary and 
monetary responsibility. 



After this exercise was over, Muriel realized that there was 
confusion among some team members about what was being asked 
in the "For what purposes" column. It appeared that some members 
thought this was asking "for what purposes" the course itself was 
being offered, rather than "for what purposes" the various 
stakeholders wanted to know particular information. 

She concluded that, in the future, this task needs to be 
explained clearly to team members, particularly because this was not 
something any of them had ever been asked to go through before. 
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Designing a strategy for gathering data 



On August 17 th, Carol and Muriel met in Albany with resource 
persons Paul Jurmo and Sue Folinsbee and three others interested in 
learning about team-based evaluation. The meeting was structured 
as a day-long workshop in which the participants would, as a group, 
help Muriel and Carol design a strategy and instruments for 
gathering the data appropriate for their sites. 

Muriel presented the lists, outlined above, which her Stony 
Brook team developed in response to the question of "who wants to 
know what information for what purpose?" 

Carol likewise presented her list. She explained that, because 
her program was not yet as developed as Muriel's, she had not gotten 
as much input from her team about the information various 
stakeholders might want. 

The workshop participants then reviewed sample instruments 
from other sites, noting which of those instruments focused on tlie 
kinds of information Carol and Muriel wanted. Carol and Muriel 
identified several data-gathering activities which they would like to 
use in the coming year. Muriel's activities are presented below. See 
the case study for SUNY Albany for Carol's list. 



Data-gathering activities which Muriel 
would Uke to use in the coming year 

Interviews with: 

~ participants in math class. 

~ their supervisors 

~ their co-workers 
(These would produce information for employees and supervisors. 
Would prefer to do this orally rather than in writing. But might 
consider a "quickie" questionnaire.) 

Designing data-gathering instruments 

As a next step at the August 17tli workshop, Muriel and Carol 
~ with the help of other workshop participants ~ prepared drafts of 
the data-gathering instruments they hoped to use. 
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They broke into the two groups, one to help Muriel develop her 
instruments and the other to help Carol. As they divided into their 
groups, participants were urged to consider the following guidelines: 

Things to consider when developing drafts of 
data-gathering instruments 

1. Clarify what category(ies) of information* you want to gather 
from the source(s) you've selected. 

2. For each category of information you want, brainstorm 
questions you want answered. Be as specific as possible. 

3. Once you have developed your questions, decide: 
~ Who would use this instrument? 

~ From whom would the information be collected? (And how 
will you select those sources/informants?) 

- When (and how often) will you use these instruments? 

- When you have collected the information, how will it be 
organized? By whom? 

- Who will analyze the information once it's organized? How 
will they do so? 

- How and when would the resulting data and analysis be 
presented? To whom? 

* "Categories of information" in this case were five types of 
information which the resource persons have identified as 
typically sought in workplace education evaluations, as follows: 



What Information Do Workplace Education Evaluations 

Commonly Want? 

1. What are the goals of tiie program? (What are stakeholders' 
expectations?) 

2. What in fact is being achieved in the program (in terms of both 
anticipated and unanticipated outcomes)? 

3. Should stakeholders continue to invest in the program? 

4. What are the strengtiis and limitations of the program? (What 
factors are helping to produce positive results? What factors are 
inhibiting progress?) 

5. What actions need to be taken to strengthen the program in the 
future? 
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The two groups then produced drafts of data-gathering 
instruments and plans for using those instruments. Muriel's 
instrument and plan are presented below. (See the SUNY Albany 
case study for information on how Carol designed her instnmients.) 



Muriel's instnimpT it and implementation plan 
Questions to ask supervisors 

1. Goal-related q uestions! 

(For those who knew about the course) 

a. What do you believe the purpose of this course was? 

b. What did you hope the course would accomplish? 

(For those who might not have known about the course) 

c. Did you know about the course this summer? 

d. What do you believe was the content of the course? 

2. Questions focusing o n anticipated and unanticipated outcomes : 

a. What have you seen as results from your workers taking this 
course? 

b. What kinds of changes have you seen in your workers? 

c. What impact has it had on your work team? 

d. Which of those results did you expect? 

e. Which results were surprising? 

3. Questions related to whether to continue the program : 
Would you recommend that other supervisors support a course 
like this? (Would you recommend to other supervisors that 
their workers attend this course if it's offered again?) 

4. Questions related to program strengths, limitations, and needed 
improvements : 

Streng ths: 

a. What was the benefit to you as a supervisor of having your 
worker (s) in this course? 

b. What, in your opinion, were the benefits for your workers (if 
any)? 
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Limitations: 

a. How could the course be improved? 

b. How could the following aspects be improved? 

• Communication about the course (to supervisors and 
workers). 

• Supervisors' involvement. 

• Scheduling/release time 

• Content of the course 

c. What would make the course more valuable to supervisors? 

d. Whsit otlier basic skills courses would be helpful for your 
workers? 

• Reading? 

• Writing? 

• ESL? 

• Oral communication? 

• Problem-solving? 

• Others? 

Implementation plan 

1. First week of September : Muriel will meet with advisory 
committee to expand on "Who, what info, for what purpose, and 
how" list. Meeting will have three goals: (1) flesh out categories, 
(2) come up with 3 drafts of instruments for feedback, and (3) 
report what classes will start in fall. Committee will develop a 
work plan, make decisions about who wiU do what. 

2. October.'93 : Will present the analysis of data to committee in 
draft report and decide how to distribute results. 

3. October.- November '93 : Will present final report. 

4. November '93 : Distribute results. 



Phase III : Gathering, organizing, analyzing, and 
reporting the information 

Gathering, organizing, and analyzing the data 

At the August 17th workshop, the resour ce persons had 
presented the participants with guidelines for organizmg, analyzing, 
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and reporting data. Referring to them, Muriel returned to Stony 
Brook to work with her team to create interview guides and 
questionnaires from the core questions developed on August 17th. 
To do so, the team referred to the "Who wants what information" 
flipcharts to identify what questions to ask of particular sources. 

The team then gathered information from four sources: (1) 
participants in the sunmier math course, (2) workers who signed up 
but did not participate, (3) supervisors of class participants, and (4) 
union representatives. 

Several team members gathered information. For example, the 
Director of Residential Operations distributed the questionnaire for 
class participants and kept track of those who returned it. One of the 
two class participants on the team served as interviewer, gathering 
data from supervisors. 

As the responses from these sources were recorded, they were 
conveyed back to Muriel. She summarized the responses on her 
word processor, then circulated them back to the team to review. 

In a meeting in late September, a subcommittee of the team 
reviewed the responses. Members coded them, cut them apart, and 
taped the responses on the wall under categories identified by the 
team. When the data had been categorized in this way, Muriel asked 
the team members to go back through each category and note any 
observations or comments they might have. She recorded their 
observations and asked them then to add any recommendations. 

Muriel typed up the team's summary of data, comments, and 
recommendations in a draft final report. 

Reporting their findings 

The subcommittee agreed that, before disseminating the report 
widely, they should first present the draft report to the other 
members of the team. Despite being a bit nervous, each 
subcommittee member presented a section of the draft report to the 
other team members. Presentations were made on the process used, 
tiie data gathered, and the subcommittee's recommendations. 

Muriel made the first of tiiese presentations, modelling how to 
make a presentation for the otiier subcommittee members. The draft 



report was well received - and adopted - by the other team 
members. 

It was agreed that the team would now disseminate the report 
to other audiences with an interest in REACH. Muriel hoped that the 
report would be well presented and well received, although this is 
something over which she could exert little influence. 



Phase IV : Deciding what happens next 

Taking follow-up action 

Based on the positive findings regarding the first course, the 
team agreed to offer a second course. This course focused on "basic 
communications for the workplace," which dealt with job-related 
written and oral communications. The team went through a similar 
version of the original process to evaluate this 20-hour course, with 
team members using modified versions of the earlier data-gathering 
activities. (One initial interest questionnaire distributed to potential 
participants was printed in both English and Spanish, although it's 
not clear what effect adding Spanish had.) The facts that the team 
didn't have to develop totally new instruments ~ and had already 
had practice gathering and analyzing data ~ made the team more 
inclined to jump into a second round of evaluation activities. 

In a January meeting, the team once again reviewed the 
collected feedback and agreed to offer a third course, this one on ESL 
The process of organizing and analyzing this second round of data 
was carried out smoothly. The director of residence hall operations 
asked his secretary to type up a summary of the data, and team 
members efficiently reviewed and analyzed each section of the data 
in the Januar>' meeting. Team members did so as if this was now 
something routine and normal. 

The team concluded from this second go-round that (1) 
learners enjoyed the instructor and the course, (2) they were now 
becoming a "learning community," and (3) they had developed 
valuable communication skills (e.g., clear and concise speech, 
listening, body language, clear writing of memos and accident 
reports, openness to geting along with others). Participants' main 
complaint: lack of parking made it hard to get to class. 
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Evaluating the evaluation 



Feedback from SIJNY management 

According to the senior management representative on the 
team, the evaluation team process was very useful. He said that he 
initially approached the program with assumptions about course 
content which turned out not to fully represent the realities and 
interests of the workers. By involving various representatives of the 
workforce, he got a better feel for their needs and interests. The 
result was that the participants felt that the program was their 
program, not just another management mandate. 

He acknowledged that the amount of time required upfront to 
put together the evaluation plan was a limitation. However, he saw 
that the time was well spent, as it produced positive results which 
are now ongoing. "You have to do the upfront groundwork." 

He also sensed that some workers might have resisted the 
initial ONA process, but they came to see Muriel not as a spy but as a 
professional trying to find out what they needed. 

He also feels that the advisory committee process fits in with 
the quality team concept now being promoted by the university. 
Other depar-ttnents on the campus are now looking into setting up an 
education program. He is advising them to take the team approach 
and - particularly - do an ONA, something he sees as "an absolute 
necessity." 

He also felt that it was useful to have a trained facilitator 
overseeing the process. He acknowledged that no one else on the 
team had the necessary background. 

Feedback from the funder 

The representative of the Governor's Office for Employee 
Relations serving on the team generally found the process to be 
useful. It not only helped to build a meaningful program at SUNY 
Stony Brook but demonstrated options which other REACH programs 
might adapt. 
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She noted that, as the team got more experience, team 
members worried less about the evaluation process itself. Their 
questions became more focused on questions vital to improving the 
program. Team members have taken increasing ownership for not 
only the process but for the education program itself. This was, of 
course, what REACH hoped for when it originally set up the advisory 
committee. 

She concurred with the senior manager's concern about the 
amount of time which the team spent to plan the evaluation rather 
than focusing more directly on implementing the course. She felt 
that this planning time could be reduced. This would be particularly 
important in other sites which wouldn't have the luxury of having a 
trained facilitator to do the "handholding" necessary to do detailed 
planning of evaluation activities. 

She felt that the faciUtator was skilled in summarizing points 
and otherwise keeping the team meetings running smoothly and on 
target. The process opened up to team members the idea of letting 
members' voices be heard. The process gave members an 
opportunity to participate. It remains to be seen whether this 
experience will have an impact on the larger organization. 

REACH will be exploring how this process might be adapted to 
other sites. As a funder, REACH needs clearer information about 
program impact across sites. It is not clear as yet how site-level 
teams might respond to this funder need. REACH might eventually 
do a centralized training of its site coordinators, focusing on needs 
and options in the area of evaluation. 

Feedback from the facilitator 

The team facilitator said that she would most definitely 
continue to use a team approach in her workplace education work. If 
she has the choice, she would prefer to work only with team- 
oriented programs. 

She found the process to be essentially sound and would make 
only minor adjustments to streamline and refine the process. She 
would like, for example, to find ways to get evaluation findings out to 
a broader audience, including potential program participants in 
particular. 
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She was pleased to find that, despite limitations of time and 
prior experience, team members were able to grasp the tasks they 
faced and carry them out. This suggested that they had taken 
ownership for the process and felt that the team gave them an 
opportunity to have their voice heard and play meaningful roles. 

She advised facilitators to be careful, on one hand, not to 
overwhelm newly-formed reams witii tasks they can't handle. 
However, on the other hand, facilitators shouldn't underestimate 
team members. If given encouragement, members are able to 
stretch themselves and try out unfamiliar tasks like interpreting 
data, jotting comments on yellow "sticky notes" and organizing them 
on the wall, using flipcharts, making presentations to a group, and so 
forth. 

She ackiiowledges tiiat someone with experience and time is 
needed to facilitate such a process. The fact that few workplace 
educators currently have such experience or time might m.ake 
ti-arisporting this process from site to site difficult. She felt that her 
own upfront iraining in tiiis process helped her avoid pitfalls. 

The facilitator also noted: 

• Given the fact that the potential clientele for the program 
included many non-native-English-speakers, extj*a efforts 
needed to be made, to have that population represented on the 
team. 

• Facilitators should also provide meaningful roles for 
members all the way through the process, not just in a few 
initiai activities. 

o Top-level management can do a lot to make or break a team. 
If they are willing to try sharing decision-maidng with others, 
the process has that much more chance of succeeding. 
Conversely, autocratic managers can block or destroy th.e team 
process> One advantage of a team: an autocratic or otherwise 
uncooperaave member can be countered by the rest of the 
team. 

• Teams should be flexible about who serves as members. If 
someone consistendy cannot attend meetings, the tearn should 
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find out why. If that member is simply too busy, he or she 
might have to delegate a stand-in or step aside and let 
someone else serve. If that member is not coming for some 
other reason, the team should ask why and try to deal with 
whatever is inhibiting participation. 

♦ Teams also need to encourage members who are shy or not 
fluent in English to participate as fully as possible rather than 
be relegated to a secondary role. 

. REFLECTIONS 

Paul Jurmo's comments: 

• Despite some initial miscommunications with management about 
her role and what the education program might achieve, the 
facilitator persisted and carefully and systematically carried out 
the ONA and evaluation process. She was fortunate to have a 
team which had the right mix of leadership, patience, time, and 
willingness to try new roles. The team, in turn, was supported by 
their state-level funder, who made it clear it was interested in 
seeing how both the education program and the team process 
worked. As the team got more experience, they became more 
comfortable with the idea of running their own evaluation and 
education program. As such, the SUNY Stony Brook case 
demonstrates many of the elements required to make a planning 
and evaluation team work. 

• REACH had hired the facilitator to help get a program up and 
running. Its policy was to hand control of the program over to the 
site advisory committee. As the Stony Brook committee now 
makes that transition, it is faced with the question of "what next?" 
That is, will the team be willing and able to carry on some version 
of the team planning process, especially with a new program 
coordinator with no prior experience with this approach? 

• The state funder is also faced with the challenge of figuring out 
whether it mig^t r.dapt parts or all of this evaluation model to its 
own evaluation needs at the state and site level. 
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SUNY at Albany 



I. PROGRAM BACKGROUND 

CSEA and GOER -- through Project REACH - agreed in fall of 
1992 to establish a workplace basic skiUs program for custodial 
workers at SUNY Albany. REACH hired workplace education 
specialist Carol Young to plan and coordinate the program. When she 
was scheduled to begin the plarming process in March 1993, Carol 
participated in a workshop conducted by Sue Folinsbee (of ABC 
CANADA in Toronto) and Paul Jurmo (of Literacy Partnerships of East 
Brunswick, NJ) on the "organizational needs analysis" process 
developed by Ms. Folinsbee. (See Phase I below.) 

Carol Young used this methodology to analyze how custodial 
workers used basic skills on their jobs. She concluded that, for their 
current jobs as presently structured, they had Umited need for 
reading, writing, and math. When she presented her findings to the 
advisory committee, it was agreed that the basic education program 
not focus so much on specific job-related uses of reading and writing 
since few such applications existed. Rather, learners would develop 
skills through practice applying them to whatever reading and 
writing tasks were of interest to them. While some of these might be 
taken from their current jobs, many of them would, presumably, 
come from contexts outside their immediate jobs. 

In early October, the 10-week program got underway. The 
curriculum followed a "whole language" approach. It focused on 
helping learners develop their own strategies for dealing with print 
rather than mastering "skills" defined by others. Classes were taught 
by two instructors from a local adult basic education center. 
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II. BACKGROUND ON THE TEAM 



Stakeholders represented on the team 

Because it is a joint labor-management effort, REACH requires 
each of its sites to be overseen by a committee composed of a mix of 
management, union, and worker representatives. This was also the 
case at SUNY Albany, where the coordinator was provided with a 
team composed of state-level representatives from CSEA and GOER, 
frontline workers from day and night shifts, a few learner 
representatives, a higher-level manager, a supervisory janitor, and 
one representative each from the Employee Assistance Program and 
SUNY's central headquarters. 

III. THE PROCESS WHICH THE TEAM WENT THROUGH 

Phase I : Facilitator prepares herself and prepares 
the team. 

Joint preparations with coordinator from SUNY 
Stony Brook 

As at SUNY Stony Brook, the "facilitator" of the SUNY Albany 
evaluation team process was the site coordinator, in this case Carol 
Young. This fact distinguished both sites from the other cases 
profiled in this study. In those other sites (Bull Worldwide 
Information Systems, Datatec Industries, Haartz Corporation, 
Nortoncompany, and Victoria General Hospital) the facilitator role 
was played by "outsiders." These outsiders had considerable 
experience with the team evaluation methodology and a significant 
investment in that approach. At the SUNY sites, the coordinators 
had an interest in the team notion, but no real prior experience using 
it. 

The two coordinators agreed that, to prepare themselves for 
their roles as evaluation team facilitators, tliey would get some extra 
up-front help. This help came in the form of on-site workshops and 
informal telephone consultations provided by outside resource 
persons Paul Jurmo and Sue Folinsbee. (For details of what 



happended in those workshops and consultations, see "Joint 
preparations with coordinator from SUNY Albany" under Phase I of 
the Stony Brook case study which precedes this one.) 



Facilitator prepares her evaluation team. 

Carol had been presented with an already-existing advisory 
committees when she was hired to plan and coordinate the program. 
As she led her team through the organizational needs analysis, the 
committee gradually began to create an identity for itself as a group 
of active decision makers rather than as passive "rubber stamp" 
groups merely giving formal approval to decisions made by others. 

By July, Carol's team had spent most of its time figuring out 
how to pull together a program incorporating the recormnendations 
of Carol's ONA. Little time was given to the question of evaluation 
other than to acknowledge that, at some point, an evaluation would 
be done. Carol even prepared an introductory exercise in which 
team members would do basic planning of tlieir evaluation, but the 
team didn't have much time to focus on those activities. 



Phase II: Planning the evaluation 

Clarifying program goals 

Despite being distracted by the tasks involved in getting the 
program up and running, Carol was in a July team meeting able to 
get them to pay attention to some evaluation-related questions. She 
got the team to brainstorm what they felt the program should focus 
on. They identified reading, writing, and GED preparation as the focal 
points of the program. They did not require that the program be 
particularly job-related. 

This limited goal statement can be compared with the more- 
comprehensive list of tentative goals which Carol had identified in 
her July 6th workshop with the SUNY Stony Brook coordinator. At 
that time, she felt that the program might: 

1. Help non-readers to read. 



2. Help participants identify educational goals and get on a path 
toward those goals. 

3. Establish ongoing computer-assisted instruction at SUNY. 

4. Establish ongoing study groups facilitated by learners. 

5. Reach every shift. 

6. Establish a model for working with REACH sites. 

7. Develop a replicable model. 

(In October, Carol noted that, despite these early negotiations 
about program goals, some team members still weren't clear as late 
as October what the program was to accomplish. This confusion, she 
felt, was due to the fact that the program had first been talked about 
as a job-related program, but the ONA had indicated that workers 
used little reading and writing on their jobs. The conmiittee had 
then agreed to let the program take a broader focus, but apparently 
some team members didn't really catch on to ~ or buy into ~ the 
ONA's findings and continued to assume that the focus would be job- 
related reading and writing.) 



Identifying who wants to know what information for 
what purpose. 



Because for much of the program cycle the team was focused 
almost exclusively on getting and keeping the program going, it 
team did not get around to the details of mapping out an evaluation 
strategy. Carol thus never got tlieir input about the question of "who 
wants to know what information for what purpose?" 

She held in reserve the tentative list she had brainstormed on 
July 6th. This tentative list was as follows: 
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Who 


What Information 


For what purposes 


• 


SUNY Central 


• Do they like class? 


• To document 
curriculum's surcps^ps 


• 


EAP Coordinator 


• Do they buy in and 


(what works). 






continue? 


• 


Upper-level mgt. 




• To decide RFACH'*; 




Director 


• Beyond this program, 


future directions. 




Asst. director 


will thpv pnntinnp with 






Union leadership 


V^JtiiCi Cdli^clliUll \>Vlicll 


^ lu Licirny p*iriicipanis 


• 


support should we 


personal goals and 


• 


Mid-level mgt. 


nrovifip?^ 

L^lWVlVi^* f 


them. 


(Dir.of Maintenance) 


• Hn <? thpir ^plf-p^fppm 

1 144.0 1.11^11 OV.1 1 L V. V. ill 








improved? 




• 


Supervisors 


P Q 

e.g., 






Custodial 


ns a Ipnrnpr 

... iA,\J C\ l^Cll llwl 






Maintenance 


^p1 f-pnn f iH pnr*p 

... o^ll V,.L/111 lviV> 1 In. Vi. 

... self efficacy 




• 


Participants 


e 2 hnndlinp strpss 

' 11C411 vAlll 1^ O LI , 

taking risks. 




• 


Co-workers 


• Can this model be 




• 


In Albany: 


"exported"? 






GOER 








CSEA 


• What are individual's 
goals? Do individuals 




• 


Ed. consultant 


achieve their goals? 





Designing a strategy for gathering data 

Carol went ahead and mapped out a tentative strategy for 
collecting the information she anticipated the team would want. She 
did so in the August 17th workshop described under "Designing a 
strategy for gathering data" in the SUNY Stony Brook case study. 

With input from other workshop participants, Carol developed 
the following list of data-gathering activities she hoped to use: 
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Possible data-gat±iering activities 
to be used at SUNY Albany 

• Observation (and videotaping) of student presentations at 
graduation. 

• Pre- and post- interviews with learners. 

• Something for a group of supervisors. They would evaluate 
program and/or clarify initial goals and follow-up at end of 
program. 

In presenting this list, Carol explained that, because her 
program was not yet as developed as Muriel's, she had not gotten as 
much input from her team about the information various 
stakeholders might want. 

Designing data-gathering instruments 

As a next step at the August 17th workshop, Carol and fellow 
coordinator Muriel Medina ~ with help from otiier workshop 
participants ~ prepared drafts of the data-gathering instruments 
they hoped to use. (For details of how Uiis was done, see "Designing 
data-gathering instruments" in the SUNY Stony Brook case study.) 

Carol held onto tiiose draft instruments for several months 
until her program reached a point toward the end of the instructional 
cycle when the team was ready to turn its attention to evaluating the 
program. At that point ~ in November ~ Carol presented to the 
team her ideas for a series of questionnaires (for students and 
supervisors) and interviews witii students. Team members reviewed 
the questions she had prepared for tiiose activities and suggested 
adjustments. With tiiat input, Carol firmed up tiie data-gathering 
activities described under Phase III directly below. 

Phase III : Gathering, organizing, analyzing, and 
reporting the information 

Gathering, organizing, and analyzing the data 

As stated earlier, as classes got underway the project 
committee focused most of its energies on organizing and supporting 
the classes. Some data-gatiiering was done in the beginning weeks 
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for the purposes of assessing learner interests. This was done 
through pre-program interviews conducted with individual students 
by instructors. 

The idea of evaluation was introduced to supervisors in a 
"sneak preview" session in October before classes got underway. 
Supervisors and potential participants were brought into the 
classroom and participated in a sample class. At that session, the 
supervisors and potential participants were told they would be, 
consulted toward the end of the program, to get their feedback. 

At the end of the instructional cycle, Carol disseminated a 
questionnaire to students and supervisors to ask for their feedback 
on what was accomplished and how the program might be improved. 
Instructors also met with individual students or pairs of students to 
get feedback via interviews. 

Learners also gave feedback to team members at the end-of- 
program graduation ceremony. Several learners made presentations 
about the benefits they had accrued in the program. One such benefit 
was having the opportunity to revise the forms they have to use on 
the job, to make them more usable. 

Carol compiled the results of these end-of-cycle activities, 
fin.ding that, in general, participants liked the class, as it got people 
into a positive learning situation while increasing learner confidence. 
This was less true for lowest-level learners, who apparently felt that 
the class was above them. In their feedback, learners also indicated 
an interest in a GED-preparation class. 

Reporting the findings 

Carol presented the results of the end-of-cycle data-gathering 
at a team meeting in January 1994. One team member questioned 
whether the feedback given by learners to instructors was colored 
by learners' reluctance to say anything critical to the instructors who 
had tried to help them. 

The team's make-up was in flux at this stage, with the learner 
on the team being replaced. This sudden injection of a new member 
dismpted the continuity of discussion about the program. 
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Phase IV : Deciding what happens next 



Taking follow-up action 

Because the findings indicated that the learners at SUNY- 
Albany might need other learning opportunities (ranging from 
tutoring for lower-level learners to GED classes), program 
administrators have arranged to have a REACH-sponsored Learning 
Resource Center available to all state employees in the area, 
including participants in the SUNY- Albany program. This' would 
allow state employees of all skill levels to pursue educational 
opportunities beyond those offered at specific worksites. 

The team also agreed to continue meeting as a team and carry 
out planning and evaluation activities in the future. At this writing, 
it was unclear just what the program would consist of in the future, 
so specifics of planning and evaluation activities were not known. 

Team members did agree, however, that in future evaluations 
team members other than the instructor would be in charge of 
intervie^ving students. This would deal with the possibility that 
learners would be reluctant to give critical responses to their teacher. 

Evaluating the evaluation 

Looking back at the above evaluation process, Carol Young 
identified a number of positive aspects of tlie team process, 
including: 

• The attempt to introduce a team planning process and to get the 
team to focus on evaluation questions was useful. It gave an 
opportunity for some members to get involved in the program in 
a way they might not otherwise. For example, one supervisor 
became the team's "workhorse," making sure tilings were done. 

• Instructors saw that Carol was interested in their opinions and 
felt free to talk with her about concerns and suggestions for 
concrete steps to improve the program. 

• The idea of inviting feedback from supervisors was a good one, 
as it involved an important category of stakeholders who might 
otherwise have been overlooked. 
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• Interviews proved to be a useful way of getting feedback from 
learners, too, better than reliance on just an end-of-cycle "smile 
sheet." 

• The team process allowed various interest. groups who might 
otherwise not have communicated due to intra-organizational 
politics to work together around a common concern. 

• Having a team member from the funding source (REACH) was 
useful in that it kept team members alert and positive. 

Carol felt that the process was, however, limited by a number 
of factors, including: 

• The fact that it was a 20-hour course, and thus no one had a lot 
of time to give to evaluation per se. 

• The make-up of the team fluctuated, with members coming and 
going due to changing management priorities. 

• Asking for feedback from those involved in the program can be 
frustrating if higher-up decision-makers are not ready to listen to 
and respond to that feedback. For example, learners became 
frustrated when they made suggestions for improving the forms 
used on the job, but administrative guidelines didn't allow those 
suggestions to be easily implemented. 

• The program had several false starts, resulting in lack of clarity 
about program objectives. The evaluation team had to contend 
with these different expectations, making it hard to focus. In the 
future, programs should be run by a team from the start. The 
team would begin with an organizational needs analysis which 
would show what needed to be done. The team would also set 
ground rules about the need to avoid arbitrary decisions, quick 
changes in membership, and other important issues. 

• Teams need to be aware that work organizations often have 
internal politics which can lead people to criticize the education 
program when they are really angry about something else 
entirely. 

Carol also noted: 
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• While the team evaluation idea and process are fundamentally 
useful, the process has to be made easy enough for members to 
really use. For example, it might not be realistic to expect team 
members -- who have limited time and expertise -- to go through 
the full process of designing data-gatliering instruments from 
scratch. 

• This approach to planning an employee education program is 
likely to work best in organizations already committed to team 
planning. 

IV. REFLECTIONS 

Paul Jurmo's comments: 

• Putting together a team evaluation in this site was not easy, as 
the facilitator was presented with a team which had not fully 
agreed on the program's purposes or on basic groundrules for 
operating as a team. The program was also conceived as a short- 
term pilot project, and the team was not prepared to invest a lot 
of time in details of planning an evaluation. 

• She was thus forced to do a lot of the work of planning data- 
gathering activities herself. The danger in such a case was that 
the activities she planned might not really capture the interest of 
other team members and they wouldn't pay much attention to the 
information generated. Despite these limitations, the facilitator 
persisted and was ~ with technical assistance from others 
experienced with team evaluation ~ able to pull together some 
evaluation activities which produced information of use to the 
team. 

• The SUNY-Albany facilitator noted that facilitators in other 
evaluation teams need to make the process as smooth (efficient, 
easy) as possible. One way of doing so would be for the facilitator 
to come prepared with basic frameworks for various instruments 
into which members can plug specific questions. 
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L Introduction 



In March and April, 1993 an Organizational Needs Assessment (ONA) was conducted at the 
Division of Residence Operations to: 

1. determine the basic skills upgrading needs of Custodial and Maintenance workers, 

2. identify other communication issues that needed to be addressed, and 

3. identify critical factors which could have an impact on the success of any basic skills 
programs offered. 

Based on the results of the ONA, the Project REACH— Stony Brook Advisory Committee 
decided that the first course offered to Custodial and Maintenance employees should be basic 
mathematics. 

The Governor's Office of Employee Relations (GOER) and the Civil Service Employees 
Association, Inc. (CSEA) contracted with BOCES 2, Suffolk County, to deliver a Technical 
Mathematical Skills course. The seven-week course was held during July and August, 1993. 
Fifteen employees completed the course. 

n. Evaluation Strategy 

In meetings in July and September 1993, the Project REACH— Stony Brook Advisory 
Committee agreed it would conduct a team-based evaluation to look at the success and the 
impact of the Technical Mathematical Skills course. 

The Committed decided to collect information from three groups of people: 

1 . participants in the class 

2. workers who signed up for the class but chose not to participate 

3. supervisors of participants in the class. 

Team members gathered information in three ways: 

• a survey of class participants 

• an informal survey of those who chose not to continue in the class 

• interviews of the supervisors of class participants 

The survey of class participants was sent out from the Division Director's office. The 
Division Director also conducted the informal survey of those who chose not to continue in 
the class. Three members of the Advisory Committee conducted the supervisor interviews. 

The information was gathered during September 1993. On September 29, those involved in 
gathering the information met to organize and analyze the findings, and develop 
recommendations. On September 30, a draft report was presented to the entire Advisory 
Committee, with the Committee making additional analysis comments and recommendations 
for this final report. 

Findings 

Following is a summary of the information gathered. (For a complete listing of the survey 
and interview responses, see Appendices A and B.) 
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Results from Survey of Course Participants 

1. What were your personal goals for the Technical Mathematical Skills class? 

Most goals were related to improving math skills. Other goals were self improvement and 
to improve how they did their job. 

2. What were you hoping to achieve that you didn't? 

Most achieved what they had hoped. Some wished the course could have been lotiger and 
that the instructor could have spend more time on fractioas and decimals. Others spoke of 
ftirther educational goals that could not be met in a single course. 

3. What did you achieve that you weren't expecting so? 

Many participants improved their math skills more than they had thought they would. 
Some mentioned specific skills such as multiplication, fractions, ratios, and solving 
problems. One participant had not expected to receive a good score in the class but did. 

4. What are the two most important things that you learned in the class? Why are these 
things important to you? 

Several participants described specific skills such as measuring, figuring square footage, 
doing multiplication, working with fractions, taking care of time sheets and checking on 
one's paycheck. One participant talked about the importance of continuing education and 
being more confident. Another felt one of the most important things was how class 
participants worked to help each other and pull together as a unit. 

5. What things did the teacher do in class that helped you learn the most? 
Participants felt that the teacher's patience and his willingness to answer questions were 
very helpful. They also appreciated his consistency and his ability to explain difficult areas 
in simple terms. 

6. What can you do now that you couldn't do before? 
Specific skills mentioned included: 

• ratios 

• measurement in inches 

• read the fractions on a ruler 

• finding the area of a room and the circumference of a circle 

• multiplication and fractions 

• new methods of solving problems 

• checking time sheets and paycheck amounts 

Others stated that they are able to compute faster and are better at applying their math skills 
to work situations. 

7. Did you learn how to apply the math skills to your job? 
Most said yes. One participant said no. 

8. Have you changed as a result of being in class? How? 

Several participants felt better about themselves and about their ability to learn. Others 
said they were able to do their jobs better. One participant commented on how the people 
in the class pulled together to help each other understand the problems. Two people said 
they had not changed as a result of being in the class. 
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Results from Survey of Those Who Chose to 
Not Continue in the Course 

The Director sent a letter to each member who had initially signed up for the course but did 
not attend any of the classes, to determine the reason. There were seven workers in this 
category. Of the five workers that responded, each indicated that they did not know the 
classes would be held on Mondays and they had planned to take annual leave on Mondays 
during the summer. 

Results from Interviews with Supervisors of Course Participants 

1. What kind of changes have you seen in your employees who completed the Technical 
Mathematical Skills course this summer? 

Although several supervisors had not noticed changes yet, others reported specific 
improvements in using measurements, both in the trades and in measuring custodial 
chemicals. Two supervisors said that those who took this math course are looking forward 
to attending other classes, and one supervisor commented that those who took the classes 
were more self-reliant and confident on the job. 

2. Are they able to apply what they learned to the job? and 3. Did it improve the employees' 
capability to do their job? 

Custodial supervisors said that participants are better able to measure and mix chemicals 
properly. Maintenance supervisors noticed that class participants were now able to figure 
out square footage and apply this knowledge to planning materials for jobs. Another 
supervisor noticed an improvement in doing time sheets. 

4. What impact has the math class had on your department/work team? 

Several supervisors noticed that the new skills and self-confidence gained made for more 
productive workers who can work independently more often. One supervisor noted that it 
had an invigorating effect on their work team, another said that the person who had taken 
the math class was now able to help others measure and plan jobs. Three supervisors 
mentioned that it had generated interest in fixture classes. Five supervisors either noticed 
no impact or did not respond to the question. 

5. What, if anything, wasn't achieved that you hoped would be? 

This question was unclear and supervisors had difficulty responding to it. Therefore, the 
Advisory Committee discarded the question. 

6. What were any unexpected outcomes? 

Supervisors had not expected the high level of interest in taking fiiture classes. One 
supervisor commented that the employee had learned more than she thought he would. 
Another unexpected outcome was seeing that the closer working relationship developed 
among some class participants carried over to the job. 

7. Do you have any suggestions about future courses? 
Suggestions for specific classes were given as follows: 

• math (5) (advanced matli and also basic math again) 

• reading comprehension (4) 

• ESL (3) 

• communications (2) 

• teach class in not only English but Spanish 

• More trade-oriented courses 303 
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IV. Analysis Comments 

After analyzing the findings, the Advisory Committee fnade the following observations: 

• The class definitely helped people in their jobs. They are applying the math skills they 
learned to their jobs. As a result, cost savings result from less waste of materials such as 
chemicals, floor tiles, and wiring. When there is less waste of materials, we are better 
able to properly stock materials. In addition, safety conditions in using chemicals are 
improved. 
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The program has captured individuals' attention and broadened their horizon. People saw 
the class work as a challenge, and they did it. Participants in die class told others what the 
class was like and, as a result, people are interested in future classes. 

The evaluation shows the importance of having a good instructor. 

At first people were quiet and didn't talk much in class. But as the class went on, people 
asked more questions. People started realizing, "Hey, it's safe to get involved here!" The 
habit of asking questions in class also spills over into asking questions in the workplace. 
The supervisors on the Advisory Committee felt that is good because, if people don't ask 
questions on the job, supervisors don't know what they need or what they are thinking 
about. * 

Learning new skills was a confidence builder. Increased self-confidence impacts other 
behaviors such as: 

• asking questions 

• helping others at work 

• willingness to try and/or learn new tasks 

When people improve their skills, it makes everyone's job go faster. The evaluation also 
showed that people are starting to work better as a team. 

There is, however, a definite cost to having that many people out for class and for two 
hours at a time. We were finding a lot of work wasn't getting done because people were 
in class. This should be less of a problem in the Fall because people will only be in class 
for one hour of work time; the second hour will be on their own time. 

V. Recommendations 

Based on the findings of this evaluation, the Advisory Committee developed the following 
recommendations. 

Fall Courses 

1. Offer a Technical Communications Course that focuses on job-related reading and writing 
skills. Offer the course for two hours, twice a week, beginning the week of October 18 
and continuing through the third week of December. One hour of each class will be work 
time and one hour will be employee time. 

2. Continue to explore interest in a job-related ESL course. If there is enough interest, the 
course could begin a few weeks after the Technical Communications Course begins. 
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Getting course information out to employees. 

1. Emphasize that the courses 2Sq job-related. 

2. Have written communications about the courses come from Fred Tokash and Phil Santella. 
This would show the union's support, encouragement, and active involvement in Project 
REACH— Stony Brook. 

3. Include examples of participant and supervisor responses from this evaluation when 
marketing the up-coming Technical Conmiunications course. 

4. Note that the ESL course (if offered) has a bilingual teacher. 

5. Supervisors could ask employees who speak both English and Spanish to encourage other 
Spanish-speaking employees to take the ESL class. 

Making sure the courses are job-related. 

1. Have the instructor meet with supervisors to get input about what should be included in 
the course and find out their current needs as supervisors. 

2. The instructor could meet employees and talk about what the Technical Conmiunications 
Course will include. She could also explore interest in an ESL course among Spanish- 
speaking employees. 

VI. Director's Comments 

First, I want to thank each of you for participating in the process to identify the types of 
training you thought were necessary to enhance productivity and increase your potential for 
promotion. I also want to especially thank those who attended our first course and, by their 
critiques, helped us to identify where we did well, and where we could improve on our next 
course offering. 

Back in early July, the Advisory Committee met to establish the goals we wanted to achieve 
with this course. The overall goal was to have the training interested and useful. Persoiial 
development goals were to improve self-esteem and make people hungry for more training. 
Work atmosphere goals were to meet the needs of Residential Operations, improve morale 
even more, create a feeling of caring, and have better employees. 

As I read the critiques from both the students and the supervisors, and compared them to the 
goals established, I think we really hit a "home run" on this program. With your continued 
support, we can keep our hitting streak alive with our next courses. 



305 



6 



APPENDIX A 

Results from Survey of Participants 
in Technical Mathematical Skills Class 

1. What were your personal goals for the Technical Mathematical Skills class? 

• 85% 

• To brush up on past skills. 

• To refresh my memory on things forgotten. 

• To tell the truth, I didn't get any special goal about it. I just thought that the course 
could help if I had to go to Maintenance. 

• To learn how to do my job better. 

• My present position involves inventory, storage, storeroom records, receiving and 
distribution, requisition transactions, etc. My personal goals were to enhance my math 
knowledge. 

• To better my math skills. 

• To improve on math that I hadn't used in years. 

• To see how much mathematics skills I need and how it can help me. 

• I went to improve my mathematical skills and better perform some of my tasks. 

• To try and improve myself. 

• I want to learn because I did not learn much schooling when I was a child because I 
was a child bom out of abuse. 

2. What were you hoping to achieve that you didn 't? 

• I hoped they could have been longer. My skills needed help. 

• My achievement was fulfilled. 

• I would like to become a surveyor. For that was my profession in my country (from 
courses taken). I cannot. But, at least I could work in Maintenance. 

• Nothing. 

• I did achieve what I was hoping. 

• I achieved what I'd hoped for and more. 

• My multiplication. 

• I am still looking to achieve high school. 

• I wish the instructor could have spent more time on fractions and decimals. 

• Refresh my mind in addition and subtraction of decunals and how to track my time 
sheet. 

• No response (2) 

3. What did you achieve that you weren't expecting to? 

• I can do the times table better and think better in class than years ago. 

• I learned a few more things from the Technical Mathematical Skills class. Then I 
reviewed fractions and others. I really didn't expect it. 

• Have a good score in the class. 

• I learned more than I thought I would. 

• I did not expect the course to present math problems such as ratios, measurements and 
dealing with improper fractions; however, I was pleased to have achieved solutions. 

• I learned about math that I couldn't figure out in school years ago. 

• I achieved how to do all kinds of math that I never had before. (2) 

• To understand a little better to solve certain problems that I thought difficult at the 
time. 

^ •No response. 
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4. What are the two most important things that you learned in the class? Why are these 
things important to you? 

• The two things I learned are (1) measurement in inches and (2) rounding off hundreds. 

• 1. That we worked to help each other understand better. 2. The instructor seemed to 
care that we learned for ourselves, fhese things are important because they pulled us 
[together] as a unit. 

• 1. The importance of continuing education. 2. Being more confident and challengmg. 

• 1. How to take care of my time sheet. 2. How to check on my paycheck. 

• I learned about fractions and multiplying and they help in my work. 

• I cannot pinpoint just two things. The whole class was important to me. 

• I learned how to do math better so I can do a better job. 

• Measurements. It relates to my job. 

• Square footage, because I like to build things in my spare time. 

• No response (2) 

5. What things did the teacher do in class that helped you learn the most? 

• Took time to explain things to me. (2) 

• He encouraged you to ask questions if you didn't understand and would help you. 

• He used some examples and he was very patient. 

• Being consistent and approaching everything in the most simplest terms. 

• Teacher was right with us all the time and answered all our questions. 

• He used some examples and he was very patient. 

• If we didn't understand a problem, he showed us another way to do it so we could 
understand. 

• He showed us how to work the problem and then he explained it to us. 

• The instructor was willing to spend extra time if you didn't understand any math 
problem. 

• About measuring the length of a room and about the measuring (?) 

• No response (2) 
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6. What can you do now that you couldn 't do before? 

• Ratios. 

• Measurement in inches. 

• Read the fractions on a ruler. 

• Have a better time understanding. 

• Compute faster, understand a great deal more and apply situations in my work place 
better. 

• Use an easier way to find an area of a room and the circumference of a circle. 

• Multiply and fractions. (3) 

• New methods of solving problems. 

• Check on my hours and money in my paycheck. 

• No response. 

7. Did you learn how to apply the math skills to your job? 

• Yes (9) 

• Some part of the math skills had to do with some job parts. 

• Yes, it was the most important thing for me. 

• No 

• No response 

8. Have you changed as a result of being in class? How? 

• Yes. Not only did I remember things taught in high school, I was also taught new 
materials. The class was very interesting. 

• Having already some skills, reading newspapers, working with my son on his 
homework, preventing me from making change. I just got a little improvement. Thank 
you! 

• Yes, I can do my job better. 

• I am working at it. I am finding myself constantly trying to improve limited space and 
be more effective. 

• I changed my thinking in the last year, but the learning skills class really helped me 
feel that I gained. I also plan to further my schooling and plan to become a paralegal. I 
know I have the time to do it because I stopped all my bad habits for a long time, and I 
will put everything I can to good use. 

• Yes. I can measure liquids and find length easier. Also, I saw how we all pulled 
together to help each other understand problems so we each could better ourselves. 

• I feel better about myself that I am trying to learn more and at my age. 

• Yes, because the tasks of (?) make you improve about the job and about the math of 
the job. 

• No (2) 

Other comments 

Some of these questions need to be thought about. I really haven't had the time and I 
apologize for the delay. 
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Results from Interviews of the Supervisors of Workers 
that Participated in the Technical Mathematical Skills Class 

1. What kind of changes have you seen in your employees who completed the Technical 
Mathematical Skills course this summer? 

Want to take more classes 

^ They are anxious to take more classes which are helpful. 

• Looking forward to attending other courses to improve himself for a better job in the 
future. 

Personal growth 

• More self-reliant, confident. 

Job-related - trades 

• He can read a ruler better and can do some jobs without asking so many questions. 

• Measures more efficiently now. 

Job-related - measuring chemicals 

• Understands the concept of mixing chemicals. 

• She has problems with multiplication and this class has improved her math a little and 
she practices more. 

No changes noticed yet 

• No basis for evaluation at this time. Supervising H-Quid for a short period of time. 

• Haven't done anything mathematical at this time to evaluate this. 

• No conmients. No feedback from employee. (4) 

2. Are they able to apply what they learned to the job? Please give an example. 
Measuring chemicals 

• When mixing and measuring chemicals (6) 

• Yes, the chemicals are measured the way they should be 

• Yes, by being able to mix chemicals properly and possibly measure a room 

Measuring - Trades 

• Yes, figuring out square footage. 

• Yes, transfers fractions into decimals. Can compute cubic areas; was not able to do this 
before. 

• Yes, can read ruler better and has more confidence 

• Yes, in knowing how much paint is needed to paint a room. 

Other 

• Improved doing time sheets. 

• Did not attend course to know what was taught. 

• No response (2) 



309 



10 



3. Did it improve the employee's capability to do their job? 

• Using now the proper amounts of chemicals to achieve the quality of work required. 

• Yes, don't have any hesitation to ask him for math solutions, cubic problems, fractions. 
Would not have asked before. 

• Yes (7) 

• No response (4) 

4. What impact has the math class had on your department/work team? 

Helped the department/team work 

• Very invigorating 

• Used less chemicals because they are being measured correctly 

• Worker gained self-confidence, new skills. Makes him a more productive worker, 
which improved entire department. 

• Able to help others measure not 

• The person is pleased with herself because she accomplished something beneficial, not 
only to herself but to her job. 

• He can work independently more often. 

Want more classes 

• Employees are looking for more classes 

• The employees are looking for further classes 

• Generated interest in future classes 

No help to department or no response (5) 

5. What, if anything, wasn V achieved that you hoped would be? 
[Bad question, unclear. Question thrown out.] 

6. What were any unexpected outcomes? 
Interest in future classes 

• Those that attended the math class have more people interested [in taking classes]. 

• This student is interested in taking future classes. 

• More people are interested in classes 

• This person wants to. take more classes and tells everyone she meets how great the 
basic math class was and the good instructor we had. 

People came together to help one another 

• a closer relationship with one another on the job. 

• Worked as a team to help one another solve a problem from the class. 

Other 

• He learned more than I thought he would. 

• No response (6) 
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7. Do you have any suggestions about future courses? 
SpeciHc class suggestions 

Math (5) (advanced math and also basic math again) 
Reading comprehension (4) 
ESL (3) 

Communications (2) 

• Teach class in not only English but Spanish 

• More trade oriented 

More classes (2) 
No response (3) 
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L Introduction 



The Tecliiiical Coa^mimications Course is the second job-related basic skills course offered 
by Project REACH— Stony Brook for Custodial and Maintenance workers in the Residence 
Operations Department. This 40-hour course was held October 19 through December 23, 
1993. Classes were held on Tuesdays and Thursdays from 11:30 a.m. to 1:30 p.m. 



n . Evaluation Process 

The Project REACH— Stony Brook Advisory Committee conducted an evaluation of the 
Technical Communications Course and used a process similar to its evaluation of the first 
Project REACH course, Tectmical Mathematical Skills Course. The Advisory Committee's 
team-based evaluation was designed to augment the information from, the questionnaire given 
to participants at the end of the course by the instructor. 

The Committee collected information from four groups of people: 

1. participants in the course 

2. supervisors of participants in the course 

3. workers who began the course but chose not to continue 

4. employees who originally indicated an interest in the course but did not register for 
it 

Committee members gathered information in two ways: 

1. surveys of course participants, their supervisors, and employees who had indicated 
an interest in the course but did not register 

2. interviews with workers who began the course but chose not to continue 

The three surveys were sent out from the Division Director's office, and responses from the 
surveys were compiled by the Director's secretary. Two Committee members, assisted by the 
Assistant Director, conducted interviews of workers who began the course but chose not to 
continue. 

On December 28, 1993, Division Director prepared a summary report of the responses from 
the employees who originally indicated an interest in the course but did not register for it 
(see Appendix A). On January 27, 1994 the Advisory Committee met to analyze the findings 
from the surveys and interviews and to develop recommendations for future courses offered 
through Project REACH— Stony Brook. 



314 



2 

IIL Findings 

Following is a summary of the information gathered through the surveys and interviews. 
(For a complete listing of the responses, see Appendices A-E.) 

Summary of Course Paiticipant Responses 

L Personal goals for taking the Technical Communications Course 

• to improve writing, reading, and communication skills 

• to learn more about their job 

• to get a better position in the future 

2. The most important things learned in the course 

• communicating better with co-workers, supervisors, and students 

• openness and participation in the class 

• becoming a better reader 

• becoming better at writing memos and other correspondence 

• dealing with stress 

• listening before speaking 

• understanding the problems and feelings of employees from other departments 

• the importance of body language in communication 

3. What paiticipants PxChieved that they hadn't expected to 

• learning to give clear explanations and get to tiie point of what they want to say 

• interviewing co-workers 

» learning to listen until a person is finished talking and then answering 

• better understanding of co-workers 

4. Helpful things the teacher did 

• made participants comfortable, treating them as equals and as adults, not children 

• knew her subject and gave clear, patient explanations 

• gave individual attention, "keeping on top of you," and showing concern that everybody 
learn what was being taught 

• explained and demonstrated new ideas 

5. How participants changed as a result of being in the course 

• understand more about one's job 

• more confident about reading and writing on the job and coimnunicating v/ith co-workers 
and students 

• have more confidence and interest in learning and self-bettemient 

• actually practice being more concise in conversatioas-understand more about one's job 
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Summary of Supervisors' Responses 

1. Changes in employees and their ability to do the job 

• course participants can communicate better with co-workers and students 

• improvement in writing skills 

• more outgoing, confident, assertive 

• happier, change in morale 

• too soon to see any changes 

• can read labels and instructions better 

• better understanding of using cleaning products 

2. Impact on department/ work team 

• communication improved 

• overall feeling of self-respect and morale has gone up 

• too early to tell 

• lost too many man-hours at one time 

• co-workers have seen improvement since employee took course 

Summary of Responses from Those Who Decided Not to Complete Course 

L Personal goals for taking the Technical Communications Course 

• to improve reading, writing, and English skills 

2. Reasons for not completing the course 

• difficulty understanding English 

• difficulties with finding parking 
» conflict with work schedule 



Suggestions for Improving Participation in Future Courses 

• Offer incentive awards 

^ Provide better \ ifoJ-mation about the course given beforehand 

• Give courses for people who are not fiuent in English 

^ Make the course location more accessible, (parking was a problem) 

• Offer same courses more than once 

• Continue getting instructors that have patience and take the time to thoroughly 
things 
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• Let people know that they will not be graded, that everyone is there to learn, and that 
learning can be mn 

• Include more writing, spelling, reading, and role-playing 

• Limit the number of participants from one work area 

• Adjust class schedule to take restroom cleaning into account 

• other courses to offer 

- a course on teamwork 

- advanced courses in math and reading 

- a basic reading course 

- a GED course 



IV. Analysis Comments 

As the Advisory Committee members read and thought about the results of the surveys and 
interviews, they made the following observations: 

Based on Participants^ Comments 

• Maybe we should ask for personal goals at the beginning of the class and, at the end of the 
course, compare them with what they learned 

• People wanted to be more effective on the job and improve themselves 

• A lot of important learning was not cited as goals, and was more specific 

• Participants appeared to become comfortable and confident with themselves and 
increasingly more eager to learn new things 

• Everyone seemed to achieve what they set out to do, and were excited about relating it to 
their jobs 

• Perhaps we should have a longer class, say 60 hours instead of 40 hours; although for 
many the class length was just right 

• There seemed to be a strong impact on worker relationships. People used words such as 
'^understanding," "get along with,*' ^get to know better" when talking about the impact of 
the course 
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Based on Supervisors' Comments 

• Many changes seem to be attitudinal in nature; morale has improved, vStaff more confident 
in themselves, which results in efficiency and pride in their jobs 

• Supervisors have gained more confidence in staff. Better mutual understanding 

• Communication is more widespread and not limited to supervisor-to-workers 

• Reading and communicating are the main behavioral applications of what people learned in 
class; morale and attitudes equally in evidence 

• People have more confidence as a result of participating in the course 

Based on the Comments of Those Who Did Not Complete the Course 

• The course was not what they expected; we need to tliink about the flyer for the next 
course 

• Staff has been given every opportunity to complete the course and must be flexible and 
grateful that these courses have been offered 

• Perhaps we should have some type of exit interview when a person drops out of the course 

• Both those who completed the course and those who did not had the goal of improving 
their reading and writing skills 



Suggestions 

• We definitely see the need for ESL 

• People want more reading courses 

• If possible, incorporate teamwork topics in future reading, writing activity in class 

Comments about the Evaluation Process 

• It's tough to bring in those with limited skills— How to do this? 

We need to respect what people tell us on the surveys, even if we don't agree with them 
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V. Recommendations 

Based on the results of the evaluation, the Advisory Committee developed the following 
recommendations: 

Evaluation Results 

1. Make the Evaluation Report available to all employees 

2. Members of the Advisory Committee should make sure the Report is circulated to all co- 
workers 

3. Prepare a short, easy-to-read summary of the Evaluation Report 

4. Make sure that key supervisors understand the importance of sharing the evaluation results 
with their staff 

5. Have supervisors hold staff meetings and read the Evaluation Report summary so 
everyone hears it 

Recruitment for Future Classes, including the ESL Class 

1. Use a picture or sketch on the course flyer to get people's attention and give an idea of 
what the course will be about 

2. Use the video that was made in the Technical Communications Course as a recruitment 
tool 

3. Have the people who were involved in making the video also be involved in showing the 
video to Residence Operations staff 

Course Levels 

When possible, offer more than one for each course, for example: 
Level 1 - Introductory content 
Level 2 - Basic content 
Level 3 - Advanced content 
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PARTICIPANT'S CRITIQUE 

1. What were your personal goals for the Technical Communications 
Course? 

*To be more effective through communications; to improve 
writing skills; learn how to make a point through short 
communication. 

*To learn better about my job; to read better. 

*The description of the course was misleading. I thought it 

was to better your reading and communication ability and it 

wasn't. Speeding my reading skills. 

*To improve on my writing correspondence. 

*My personal goal was to learn more about the work rules and 
communications. ^ • 4. • 

*To write better memos. Trying to use the right pronunciation 

when I speak. i 

*To brush up on my communication skills; to better learn ro 

communicate effectively; to be able to communicate with as few 

words as possible. , , 

*To learn more about the job and communication with the 

students. , 

*To learn where to go and how to do things by going to the 

proper mail as to the right person. 

*I wish a better psoition in the future. 

* Improving spelling and reading. ^ , 

*To write, speak and become active as far as explaining and 

actually showing someone what needs to be done. 

*To get better with my job. ^ 

2. What are the two most important things that you learned in the 
class? Why are these things important to you? 

*Classmates become more open to each other. Role 
playing/ video was made. It opens the opportunity for everyone 
to increase their potential. 

*To become a better reader and helped me find different word 
meanings. 

*How to write memos to communicate with other departments. 
*How to deal with stress. Also how to write and comprehend 
better. I feel better about myself with written 

correpsondence . 

*I learned how to report accident and now what to do if one or 
my co-workers has an accident. I learned Stony Brook's 
regulations. 

*I leaned about communications and work rules. 

*To listen before I speak. I have a better understanding of 

how to write memos now. 

*One should communicate with others on the job in simple 
terms. This is important to me because I sometimes have a lot 
of information to relay and if I relay it simply it is 
remembered- Body language is communication. I'll try not to 
show disappointment through body language when dealing with 
clients . 
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*1 learned more about the finer details of the job. 
♦Communication through the tone of your voice and how to 
communicate without offending others. 

★Better communication with the supervisor and the partners. 
Good relationship in my partners and friends. 
♦Understanding the problems and feelings of employees from 
other Departments in relation to their jobs; it makes aood 
compromise. ^ 
*To openly participate and to ask questions when I don't 
clearly understand. it is important to me so that I am 
understood. 

*I am learning to read and for my family and Stony Brook. 

anything that you were hoping to achieve that you 

didn't? 

*I feel I achieved more than I expected. 
*I wish the class was longer. 
♦Speed my reading, 

*No, because everything was done according with the program. 
*No. 

*Yes. How to express myself to others by using body signs and 
tones of your voice. 

*My hope is to learn to vnrite better English. 

*More practice with the actual usage of English. Our class 

time never seemed to be long- enough. 

*More words and sentences. 

4. What did you achieve that you weren't expecting to? 

♦Interviewing my co-workers, which was an exciting learning 
experience. ^ ^ 

*My reading of different word meanings. 

*I was able to work with some of the slow speaking students. 
♦Instructor took the time to answer any questions about 
writing and showed me how to get directly to the point without 
prolonging it. f * uu 

*How different the co-workers are when you get to know them. 
*Some sentence consruction. 

anlwer!"^ ^° ^^^^ ^ person is donb talking and then 

♦Achieved the ability to get my point across in fewer words 
than I previously had to use. 

*I really enjoyed the class and also got a lot out of it. 

*How to get along with everybody. 

♦Progress in my work. 

♦Better understanding of others. 

^i^® clearer explanations and to be more to the point. 
♦The teacher gave me more ideas. 
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5. What things did the teacher do in class that helped you to 
learn the most? 

*Ongoing interaction, keeping on top of you and showing lots 
of concern. 

*Help me sound out words and break them up. 
♦Explained memo patterns. ^+.„^v 
*She was understanding and helped when we were stuck on 
anything. She taught us not as children but as adults. She 
made it interesting enough that I listened intently so as not 
to miss anything. 

*Her big knowledge. , , ^ , -i 

*Being patient with the students. Made us all feel equal. 
Explained everything till we all understood. Made all 
participate in the program. 

*Teacher demonstrated ideas that she wanted class to remember 

and repetition. j. +. 

*Using the blackboard and correcting me on how to use correct 

and proper phrase. 
♦Communication with others. 

*Made me feel comfortable and not to be embarrassed. 

*Gave individual attention and took time to explain to 

everyone. 

6. Did you learn how to apply the communication skills to your 
job? 

*Yes, I try to use them daily in the work place. 
*Yes, to give clear instructions and explanations to my co- 
workers. 

*I improved more. 

7. How have you changed as a result of being in the course? 

★Constantly wanting to learn more and having the opportunity 
of making it a daily challenge. 

*I learned to understand more about my Dob, feel more 
confident about reading labels and my writing has improved. 
*No, because my trade is completely technical. 
*I am more prepared to deal with the students and co-workers. 
*Being 64 I don't know if I can change but I sure will try my 
hardest to use the knowledge I have learned in this program. 
*I'm more aware of the importance of saying things m a short 
effective way and I actually practice being concise in my 
conversations . 

*I can do my job better now. 

*I changed my attitude toward co-workers and others around me. 
*Taking more time in writing. 

*I think I have more interest in betterment and confidence. 
*It made me more aware of just how important clear 
communications are. 
*To get along with people. 
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What suggestions do you have for improving participation in 
the course? 



*Early planning 

♦Incentive awards 

*Same class given more than once. 

♦Classroom at a smaller scale. 

♦Class should be a non-English speaking class; a slow reading 
class; a well reading/speaking class not all mixed together. 
♦Felt misled by the flier, maybe in the future the notices 
should be worded differently. 

♦To offer the same course to new participants, or to everybody 
that wants to take it. 

♦To come up with something that the majority can like: E.S.L. 
m different levels. 

♦Keep getting teachers that have patience and to take time to 
explain the situation at hand in reading, vocabulary and 
writing . 

♦Having a special group for the people who can't speak English 
fluently. in that group have someone who can understand 
Spanish, and can speak English well enough to help people who 
have poor English skills. 
♦Give more courses. 

♦Have the course meet in an area more, accessible for 
participants to park. 

♦More writing, spelling and reading in class. 
♦I think there should be some more role playing. 
♦Participants should know that you are not graded; that each 
one IS there to learn; that learning can be fun. You are 
treated as an adult. 

♦To learn more and improve to help Stony Brook. I like Stony 
Brook very much. We shall overcome some day. 
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snPERVISORS ' CRITIQUE 

What changes have you seen in your employees who coiupleted the 
Technical Communications Course? 

★Change in morale. w 4. 

*At this time, I have not seen any changes. It may be too 

soon to tell. , , ^ 

★Employees enjoyed course but felt there should be more. 

There was no improvement. 

★Better oral and written skills. 

★More assertive in working with co-workers. ^ 

★Improvement in reading, better vocabularly skills. Employees 

more eager to apply to job. ^ ^- ^ 

★Since taking this course participant wants to further reading 

and writing, etc. 

★More authoritative and asserts self better. 

★Yes, but needs more classes on reading comprehension. 

Are they able to apply what they learned to the job? Did the 
course improve their capability to do the job. Please give an 
example. 

★Yes, can better communicate with others. ^ 
★Learned to read the labels ^better. Better conversation with 
co-workers . 

★Too soon to answer this question. 

★Needed no improvement; employee know how to do paper work. 
★More outgoing. 

★Yes, reads labels more carefully. ^ 

★Employees have better understanding of using everyday 

cleaning products in a more sufficient manner. 

★Better communications with others. More careful with reading 

instructions. 

★Yes; much happier; improved to become a Janitor. 

★Yes, most of the time; Supervisor can call on employee for 

assistance more. 

★More assertive . 

★Yes; this course gave employee more confidence. Employee is 
able to communicate better with students. 

What impact has the Technical Communications Course had on 
your department/work team? 

★Too early to tell. 
★Communicates better. 
★None as of yet. 

★Lost too many man hours at one time. 

★Wants to go further in education. ^ 4. 4. v 

★Employees have better understanding and are willing to taKe 
charge when ordering supplies, measuring chemicals, and 
overall feeling of self-respect and morale has also gone up- 
★Communications are improved, 
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*Co-workers have seen an improvement since emplcvee took this 
course. 

i 

Do you have any suggestions about future courses? \ 
*None 

*Limit participants from one area* 

*Run a course on working as a team^ Have advance courses in 
math and reading. 
♦Upgrade math classes. 

*An indepth reading comprehension class. 

*A course on working on the values of teamwork, 

*More reading and comprehension; possibly an easier coursie 

for employees with a problem reading. rsiipervlsor 
appreciates the instructor taking an extra one-half hour to 
help this emplo yee. it has made an impact on his lifeo) 



Appendix C 



CRITIQUE OF THOSE WHO DECIDED NOT TO COMPL ETE COURSE 

What were your personal goals for the Technical Coitonunications 
Course? 

*I thought the class was going to be in improving my reading 
and writing. 

*My goals were to improve my reading and writing skills. 
*To improve my English. 

Why did you decide to not complete the Technical 
Communications Course? 

*I was having trouble understanding English and the teacher 
went too fast for Ime. 

★Difficulties in parking, confli9cted with daily work 
routines. All restroom cleaning is done on Tuesdays and 
Thursdays. Would be better if shift was changed to 6:00 AM - 
2:30 PM or 7:00 AM - 3:30 PM. 

★Course didn't cover what I though it would. I later heard 
that the course improved, Maybe I should have continued. 
★Wasn't the class that I was expecting. 

★Car troubles. Class was hard to keep up with in 
understanding. 

What suggestions do you have for improving participation in 
the next Project REACH course? 

★Change the days of class, schedule for cleaning restrooms 
gets in the way. 

★More basic reading, writing courses. 

★Prep course for GED so I can obtain a high school diploma. 
★To bring us more facility. 

★Instructor needs to slow down in her teaching, went too fast 
for me. 

QUESTIONS FOR THOSE WHOSE NATIVE LANGUAGE IS NOT ENGLISH: 

Project REACH is planning to offer a job-related ESL course 
for Custodial and Maintenance workers. Are you interested in 
taking this course: indicate yes, no or possibly. What should 
this course cover? 

★Yes, reading and writing. 

★Possibly, reading, writing and grammar. 
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Responses of Those Who Indicated an Interest 
in the Course but Did Not Register 

Summarized by 

Director of Residential Operations 



Prior to the selection of the Technical Communications Course, we sent a letter to each of 
our eniployees as to what course(s) they might be interested in, as a follow-up to our 
Technical Math Skills Course. They were offered a choice of four possibilities: Basic Math, 
a continuation of the Techriical Mathematics Skills course, English as a Second language, or 
Technical Communications. The responses indicated: 

7 - Basic Math 

7 - Intermediate Math 

10 - ESL 

25 - Technical Communications 

The Advisory Coiirmiittee selected the Technical Communications Course as our next 
offering. Prior to the start of the class, a letter was sent to each employee that indicated an 
interest in the class, letting them know that the course would be offered, when, and where. 
A flyer was also distributed to pique the interest of our other employees. 

Our initial registration included 26 students, of which 16 were from the list that originally 
indicated an interest in the course. I sent a letter to each of the nine employees that 
originally indicated an interest in the course but failed to register, to get feedback on what 
changed their minds or what we should/could have done differently which would have 
allowed them to register for the program. 

The results were mixed: 

• Four people stated that they did not know the dates for the class (This was surprising 
because all employees received a personal letter and there were flyers on the program in 
each work area). 

• Two people thought the offering would be too basic. 

• One person was going to retire within a few months and didn't want to take a seat from 
someone who would be around for a longer period of time. 

• One person had to clean bathrooms on Tuesdays, 

• One person had a parttime job on Tuesdays, 



Evaluation of Technical Communications Course 

Responses from Participants in the Course 



Personal goals for taking the course 
• to improve writing, reading, and communication skills 
• to learn more about their job 
• to get a better position in the future 



The most important things participants learned 

• communicating better with co-workers, supervisors, and students 
• openness and participation in the class 

• becoming a better reader 

• becoming better at writing memos and other correspondence 

® dealing with stress 

* listening before speaking 

• understanding the problems and feelings of employees from other departments 
the importance of body language in communication 



What participants achieved that they hadn't expected to 
o learning to give clear explanations and get to the point of what they want to say 

interviewing co-workers 
• learning to listen until a person is finished talking and then answering 
• better understanding of co-workers 



HeSpfui things the teacher did 
• made participants comfortable, treating them as equals and as adults, 

not children 

• knew her subject and gave clear, patient explanations 

* gave individual attention, "keeping on top of you," and showing 

concern that everybody learn v^/hat was being taught 
» explained and demonstrated new ideas 

How people changed as a result of being in the course 
• understand more about one's job 

• more confident about reading and writing on the job 
and communicating with co-workers and students 

• have more confidence and interest in learning and self-betterment 

• actually practice being more concise in conversations 
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Evaluation of Technical Communications Course 

Supervisors' Responses 



Page 2 



Changes in employees and their ability to do the job 
• course participants can communicate better with co-workers and students 

• improvement in writing skills 

• more out-going, confident, assertive 
• happier, change in morale 

• too soon to see any changes 

• can read labels and instructions better 
•better understanding of using cleaning products 



impact on department/work team 
• communication improved 
• overall feeling of self-respect and morale has gone up 
• too early to tell 
• lost too many man-^hours at one time 
• co-workers have seen improvement since employee took course 



Responses from Those Who Decided Not to Complete Course 



Personal goals for taking the Technica! Communications Course 

* to improve reading, writing, and English skills 



Reasons for not cor^pSeling the course 

• difficulty understanding Engiish 

• difficulties with finding parking 
• conflict with work schedule 



February 18, 1S34 
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Evaluation of Technical Communications Course Page 3 

Suggestions for Improving Participation in Future Courses 



• Offer incentive awards 
• Provide better information about the course given beforehand 

• Give courses for people who are not fluent in English 

• Make the course location more accessible, (parking was a problem) 
• Offer same courses more than once 

• Continue getting instructors that have patience and 

take the time to thoroughly explain things 

• Let people know that they will not be graded, 
that everyone is there to learn, and that learning can be fun 

• Include more writing, spelling, reading, and role-playing 

• Limit the number of participants from one work area 

• Adjust class schedule to take restroom cleaning into account 

• other courses to offer 

- a course on teamwork 

- advanced courses in math and reading 

- a basic reading course 

- a GED course 



February 18, 1994 



TFAM FVATIJATION 
AT VICTORIA GFNFRAT HOSPITAL : 
A CASE STUDY FROM A WORKPLACE EDUCATION 

PROGRAM 



Prepared 
by 

Paul Jurmo 



One of seven case studies in 
" Team Evaluation ; 
Case Studies from Seven Workplace Education Programs," 
a report prepared in June 1994 
by Laura Sperazi and Paul Jurmo 
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VICTORIA GENERAL HOSPITAL 
Halifax, Nova Scotia 



I. PROGRAM BACKGROUND 

In late 1992, Victoria General Hospital (VGH) - with partial 
funding and technical assistance from the Nova Scotia Department of 
Education (NSDE) ~ had run an employee basic skills program for 
two years, having expanded from six employees from one 
department to twenty from eight departments. 

The program takes a "learner-centered" approach, focusing on 
personal learning goals negotiated by the participant with the 
instructor. These personal goals vary from learner to learner, and 
break down into: 

. Academic goals (e.g., "earning my GED," "improving my writing"); 
Self-confidence (eg., "feeling more relaxed"); 
Tob-related goals (e.g., filling out forms, qualifying for other jobs); 
Non-iob-related goals (e.g., helping their children, further 
education). 

These learner goals were summarized as follows: 

"To enable participants to develop self-esteem, reading, writing, 
oral English, and math skills needed to solve problems they 
identify as of personal interest to themselves." 



11. BACKGROUND ON THE TEAM 
History of the team 

From its inception in fall of 1990, the program had been 
overseen by a project team composed of representatives from 
hospital management, one of the hospital's unions, and the NSDE. In 
1991-92, two program participants joined the team, as well. 
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In early 1993, the team approached ABC CANADA, a national 
foundation based in Toronto whose services include providing 
technical assistance to workplace education programs. VGH asked 
ABC for guidance on setting up a program evaluation, to document 
the program's positive impact and thereby help ensure ongoing 
support from the hospital administration. 

By coincidence, ABC's workplace consultant. Sue Folinsbee, had 
already begun preparation of a collaborative evaluation methodology 
which ABC hoped to field-test in one or more sites. This evaluation 
project was being developed with Paul Jurmo, who had already 
begun a similar project in the United States with I^ura Sperazi, 
under National Institute for Literacy funding. 

In response to VGH's stated interest in evaluation. Sue 
Folinsbee and Paul Jurmo agreed to "use" VGH as a test site for the 
collaborative evaluation methodology they were developing for 
Canada and the U.S. In turn, it was hoped that VGH would not only 
produce a meaningful evaluation but - for the longer-term ~ set in 
place an ongoing evaluation system. 

Stakeholders represented on the team 

In a meeting on March 29, 1993, the project team was ~ for 
the purposes of this project ~ organized with the following nine 
representatives of program stakeholder groups: 

Hospital management representatives : 

• Two from the Training and Development Department: 

~ An evaluation specialist. 

~ The education program coordinator for 1992-93. 

• Three from the eight departments sending workers to the 
program: 

~ One from Food and Nutritition Services 
~ Two from Modem Cleaners (a contractor to Housekeeping) 
Union representative : 

• One from the Canadian Brotherhood of Railway, Transport, and 
General Workers, one of the hospital employee unions. 
Program Darticipant representative : 

• One, an employee in Food and Nutrition Services. 
Education provider representatives : 
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• Two from the Literacy Division of the NSDE. (Both had, in earUer 
years of the program, served as instructors.) 



III. THE PROCESS WHICH THE TEAM WENT THROUGH 

Phase I : Facilitators prepare themselves and 
organize the team. 

Initial preparations 

In this case, the "facilitators" were Sue Folinsbee and Paul 
Jurmo, workplace education specialists who ~ as noted above - had 
agreed to provide technical assistance to VGH. They prepared 
themselves by: 

• Interviewing the VGH program coordinator by telephone : They 
clarified program status, team make-up, and why VGH now 
wanted to do an evaluation. 

• Asking proiect team members to complete a questionnaire : 
Members specified what help they needed, what they had already 
done vis-a-vis evaluation, who was involved, and what kind of 
information they might want to gather in a future evaluation. 
This helped the facilitators understand the team's evaluation- 
related experience and concerns, while getting team members in 
the mode of thinking about evaluation. 

Organizing the evaluation team 

Until early 1993, the team's role had been largely an advisory 
one, responding to requests from education staff. It had not to that 
point engaged in any kind of formal evaluation activities. 

To help the team to begin thinking about what they might want 
out of an evaluation, the coordinator talked with members in two 
team meetings about what evaluation meant, why do it, and the 
steps and responsiblities involved. These discussions ensured a basic 
understanding and buy-in on the part of team members. Team 
members then filled out the above-described questionnaire sent 
them by the facilitators. By carrying out these discussions and 
completing the questionnaire, members were thus oriented to their 
new roles as evaluators. 
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The facilitators prepared a plan for their first meeting with the 
team, based on the results of the questionnaires. The education 
coordinator reviewed this plan and explained the meeting's agenda 
to the team prior to the meeting. 



Phase II : Planning the evaluation. 

Clarifying program goals. 

It was originally thought that, in a single meeting on March 
29th, the facilitators could help the team analyze its education 
program and prepare an action plan for additional evaluation 
activities. Time did not permit all of those objectives to be met in 
one session. Instead, the team: 

• Clarified who the program's stakeholders were . Stakeholders 
included not only the groups represented on the team but higher- 
level management and learners' families and communities, too. 

• Clarified program goals (i.e.. stakeholders' expectations) . The 
team sunmiarized a mix of academic, self-confidence-related, job- 
related, and non-job-related objectives. They synthesized those 
goals into the following goal statement: 

"To enable participants to develop self-esteem, reading, 
writing, oral English, and math skills needed to solve problems 
they identify as of personal interest to themselves." 

• Clarified to what extent those goals have now been met . 
Referring to what they said in the earlier questionnaire, the team 
developed a list of positive outcomes already achieved by the 
program. These included examples of many of the hoped-for 
academic, self-confidence, job-related, and non-job-related 
outcomes, as well as the following unanticipated outcomes: 

~ increased supervisor interest in worker education, 
~ a new kind of "benefit" for workers, 
~ improved management-employee relations, 
-- improved understanding across departments. 
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~ learner-written materials now used in other adult education 
programs, 

— expanded tutor skills and relations with learners, and 

— strengthened educators' expertise and professional relations. 

• Clarified what evaluation activities they had used so far . The 
team identified the following as activities they had used for 
evaluation purposes: feedback from participants, learner self- 
assessment, GED exam, team discussions, formal written reports, 
review of attendance figures, and observation of learners in 
various program activities. 

• Identified lessons learned from those evaluation activities . The 
team identified lessons which the above evaluation activities had 
taught them about (a) the program's overall strengths and needed 
improvements and (b) steps needed to improve the evaluation 
system used to date. The latter evaluation-related improvements 
included: 

~ Document why the program should be continued/re-funded. 
~ Get more input from learners. 
~ Clarify why people drop out. 

~ Focus on outcomes for the institution (e.g., increased 
participation by workers in "quality" activities). 

~ Clarify how the program has changed over time (e.g., why 
enrollment is lower this year). 

~ Find out if "graduates" might want to be involved or help 
aferward. 

~ Ask learners for feedback on course content and what they 
might want to do next. 

— Involve managers and supervisors more actively in defining 
what information they need from the program. 

— See our evaluations as a way of adding to the body of 

knowledge for the workplace education field (e.g., NSDE). 

Identifying who wants to know what information for 
what purpose. 

In the second half of the March 29th meeting, team members 
began mapping out a strategy for future evaluation activities. They 
did so by responding to the question of "who wants to know what 
information for what purpose?" 
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In the first case, the "who's" for the evaluation (i.e., the 
audiences who want information) were divided into "primary 
audiences" (i.e., the President's Executive Council, prospective 
students, program staff, NSDE, VGH department heads, union 
representatives, and current participants) and "secondary" audiences 
(i.e., the print shop, audio-visual services, and training and 
development department which support the program; and tutors). 

To answer the second part of the question (i.e., "what 
information" is needed), the team reviewed the responses they had 
given to the pre-workshop questionnaire and agreed that the 
following list represents the questions they wanted to answer in this 
evaluation: 

"What Information" Do We Want Now? 

Establishing the need: 

• Is there a need for employee upgrading? 

Clarifying what was accomplished: 

• What do stakeholders see as benefits of the program? 

• Are participants moving toward meeting their goals? 

• Does management see changes in participants' attitude, 
flexibility, or confidence which could lead to improvement in 
job performance? 

• What was useful? 

• How valuable is the program? 

• Is there any way of showing how increased employee 
education can in any way be related to improved quality 
care? 

• Does the union see any benefit or improvement from their 
perspective? 

Deciding whether to continue: 

• Do we continue the program? 

• Should it be an ongoing program? 

• How can we justify continued funding? 

Deciding how to strengthen the program in the future: 

• Are we doing the right things to address the 
problems/opportunities/ the best approach to delivery? 
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• If the program is meeting student expectations, how can it be 
improved? 

• How improve the overall planning and implementation 
process, to increase participation and ensure more effective 
planning of future programs? 

• What do we repeat? 

• What do we leave out next time? 

• How does the project team feel about the quality of its own 
work as a team? 

The team members then answered part 3 of the question (i.e., 
"for what purpose" will this information be used?): 

"What Purposes" 
Will the Above Information Be Used For? 

• To ensure program improvement/efficiency : to help us optimize 
the "fit" between our goals and actual outcomes 

• To ensure ongoing support for the program : to ensure financial 
resources from funders, as well as buy-in, cooperation, and 
investment from stakeholders. 

• To model a "continuous improvement" approach to management : 
By critically analyzing our education program, stakeholders will 
get practice in team planning skills they can use back in the larger 
organization. 

Designing a strategy for gathering data 

At the March 29tli meeting, time ran out before the team could 
proceed much furtlier with, designing the evaluation strategy. 
Subsequently it was agreed that the facilitators would return for a 
second meeting, to help the team pull together specific tools and a 
longer-term evaluation strategy. 

Between the two m,eetings, the facilitators prepared minutes of 
the March 29th meeting and circulated tliem to the team members to 
reinforce what was learned in that meeting. 

The facilitators tlien prepai'ed a plan for tiie second meeting, 
along with an action plan outUning ideas for a longer-term evaluation 
project. 
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On April 19th, the team met with the facilitators a second time. 
Rather than look backward (as was the focus in the first meeting), in 
this second meeting the team was much more "action"-oriented, 
focusing more clearly on specifics of planning an evaluation. To get 
this planning going, the facilitators presented their action plan, which 
proposed the following four phases of evaluation activities, to be 
carried out through December of that year: 

Phase I : Looking back at what has been achieved to date (April- 
June 1993) 

Step 1: Develop drafts of data-gathering 

instruments at 4/19/93 meeting 
Step 2: Prepare to use instruments for 

first phase of data-gathering 
Step 3: Use the instruments to gather the data 
Step 4: Organize the resulting data 
Step 5: Analyze the data 
Step 6: Prepare a final report 
Step 7: Present the final report 
Step 8: Take follow-up action 

Phase II : Preparing for more-comprehensive data-gathering in the 
fall (June-August 1993) 

Phase III : Collecting data in fall 1993 (September-November 
1993) 

Phase IV : Analyzing the data collected in the fall (November- 
December 1993) 

The team agreed to adopt this plan, noting that details of 
Phases II, III, and IV would be planned when results from Phase I 
were in. 

Designing data-gathering instruments 

To design the data-gathering instruments to be used in Phase I, 
the team first considered three questions: 

1. What are possible sources of the information we want to 
gather? 
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2. What activities might we use to gather information from those 
sources? 

3. What are the advantages and disadvantages of each of those 
data-gathering activities? 

In a large group, team members responded to the first two of 
those questions. As possible sources of information they identified 
current program participants, potential participants, management 
representatives from eight participating departments/floors, 
participants' co-workers, instructors and tutors, program "dropouts," 
and documents. As possible ways of gathering information 
from those sources, they cited interviews, questionnaires, focus 
groups, review of documents, observation, anecdotal evidence, 
simulations, and learner portfolios (containing participant journals, 
checklists, sample writings, and other evidence of learner progress). 

Breaking into two groups, team members analyzed the pro's 
and con's of each of the above data-gathering activities. They said 
that, optimally, instruments should provide efficient collection of 
information from those who know the program, reinforce 
communication links among stakeholders, and respect confidentiality. 
Data-gathering activities should not intimidate respondents, intrude 
on busy schedules of both information source and information 
gatherer, breach confidentiality, allow bias to influence how data are 
gathered or analyzed, or overlook the full realities of the program, 
and informants' thinking. 

The team now better understood the data-gathering options 
open to them, and then proceeded to select which instruments they 
would use and to prepare working drafts of those instruments. They 
agreed that, whichever activities the team selected, the instruments 
should be relatively easy to learn and use in the time available and 
should focus on the information needed by the identified audiences. 

To be sure they would gather the information needed most 
immediately, the team agreed tiiat they would collect information to 
answer the following questions: 

Information Needed for Phase I 

Outcome-related information 

• What has been accomplished to date? 
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• Should the program be continued/re-funded and, if so, at 
what level? 

Program-improvement- related information 

• What have been the goals of the program so far (for both 
individuals and for the organization) and what might they 
be in the future? 

• What actions need to be taken (and by whom and when) 
to improve the program? 

The team agreed that, in the first phase of the pilot project, the 
following data-gathering activities might be used. (Given the limited 
time the team had to give to this data-gathering, these activities 
were listed in order of priority.) 



Data-Gathering Activities for Phase I 

1. Focus groups and interviews for current and past 
participants. 

2. Focus groups for supervisors (from eight departments) 
having contact with participants. 

3. Focus groups and interviews for those who chose not to 
participate before or during program. 

4. Interviews with potential participants. 

5. Interviews and focus groups with education staff. 

To help those team members who had never designed a data- 
gathering instrument before, the facilitators displayed a sample 
instrument, a guide for a focus group to be conducted with 
supervisors. The facilitators noted that the questions asked in that 
focus group would be essentially the same questions asked of other 
sources participating in focus groups or individual interviews. The 
facilitators explained that this consistency of questions is important 
because it allows the team to compare what various sources say 
about the same questions. 

The sample guide for a focus group for supervisors was as 
follows: 
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Sample Focus Group 
for Supervisors 

1. Individually, supervisors rate how well t±ie program has 
achieved its overall goal on a scale of 1 to 10. (1= "not very 
well," 10 = "extremely well") 

2. In small groups, supervisors come to agreement on #1 with 
reasons for their rating. 

3. Small groups then answer the following "outcome-related" 
questions. 

a. What kinds of changes have you seen in your employees? 

b. What impact has the program had on your department? 

c. What, if anything, wasn't achieved that you hoped would 
be? 

d. What were any unexpected outcomes? 

4. A spokesperson from each group presents to the large group 
a summary of the small group discussion on #2 and #3 above. 
This discussion is recorded on flipchart paper by a facilitator. 

5. The process for #3 and #4 is repeated with "program 
improvement" questions. (See below.) 

a. What are the strengths of the program? 

b. How could the program be improved regarding: 

• Content and topics 

• Teaching style 

• Time (scheduling) of program 

• Release time 

• Location/facilities for program 

• Support services (extra counselling . . . ) 

c. How could the program better recruit new students, 
advertise the program, and retain current learners? 

d. Should the program be continued/re-funded? 

e. Should the program be expanded? If so, in what way(s)? 

The team reviewed this sample guide for a focus group for 
supervisors, suggesting revisions in wording and asking for 
clarification. It was agreed that these basic questions would, as 
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much as possible, be adapted for otlier data-garhering activities to b^^ 
carried out by the team. 

In two groups, members prepared drafts of guides for the 
other focus groups aiid individual interviews the teajrn had eyxiier 
agreed to carry out. They reconvened and displayed tiie dxdit. 
wording of their instruments on flipchart paper. The team agreed on 
working versions to be used when they next actually went out and 
gathered data. 

The facilitators took some time at this point to raise the 
following issues fcr the teajn members to keep in rnind as their 
evaluation work proceeded: 



Issues Needing Further /Attention 

1. When do we use sampling? And how do we do it? 

2. How do we develop necessary interviewing skills? 

3. What do we do with supervisors who don't know which of 
their employees ai'e in our program? How do we maintain 
learner confidentiality in such a case? Do we ask learners 
first? 

4. What options are there for those who can't attend a focus 
group? Do we leave them out? Do we try to arrange a second 
focus group and/or individual interviews for them? 

5. How do we identify "informal leaders" among potential 
participants to be interviewed? 

6. When interviewing education staff and tutors, be sure to ask 
them to identify personal outcomes they have achieved by 
working with the program. 

7. Avoid using individual names, to protect learner 
confidentiality. 

After discussing those issues, the team agreed who would be in 
charge of using which instrument, when data would be gathered in 
the next 1-2 months, and to whom the resulting data would be given. 
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Phase III : Gathering, organizing, analyzing, and 
reporting the information 

Gathering and organizing the data 

During the period of May through July 1993, team members 
used interviews, focus groups, and questionaires to collect 
information from 43 informants, including 19 program participants, 
four "early terminations," three instructors and two volunteer tutors, 
seven supervisors, and eight members of the planning team itself. 

As members realized the difficulty of organizing so many data- 
gathering activities with limited time and personnel, they pared 
down the data-gathering schedule accordingly. 

Those using the instruments summarized the responses and 
relayed them to the project coordinator. She recorded the responses 
in writing and on her computer, and then organized them by cutting 
and pasting them into meaningful categories. 

She took pains to be both concise and detailed enough to reflect 
what the informants actually said. She also tried to avoid inserting 
any "editorial comments" of her own. If such comments came to 
mind as she recorded the informants' responses, she noted them in a 
separate space titled "coordinators' interpretations." (See "Analyzing 
the data," below.) 

With long-distance guidance from the facilitators via telephone, 
the coordinator laid out an outline into which she could insert the 
data. The key categories of the outline were the four major 
questions which the team had set out to answer. Under each of those 
questions, she had sub-categories organized by data source. The 
resulting outline looked like this: 

Outline for organizing the collected information 

1. What have been the goals of the program ? 

a. What learners said 

b. What "discontinued" learners said 

c. What education staff (both paid and volunteer) said 

d. What management/supervisors said 

e. What the project team said 
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2. What has been accomplished to date ? 

a. What learners said 

b. What education staff (both paid and volunteer) said 

c. What managers/supervisors said 

d. What the project team said 

3. Should the program be continued and whv ? 

a. What learners said 

b. What education staff (both paid and volunteer) said 

c. What managers/supervisors said 

d. What the project team and discontinued learners said 

4. What actions need to be taken to improve and expand the 
program ? 

• Course content and teaching style 

a. What learners said 

b. What education staff (both paid and volunteer) said 

c. What managers/supervisors said 

d. What the project team and discontinued learners said 

• Location 

a. What learners said 

b. What education staff (both paid and volunteer) said 

c. What managers/supervisors said 

d. What the project team and discontinued learners said 

• Scheduling/time 

a. What learners said 

b. What education staff (both paid and volunteer) said 

c. What managers/supervisors said 

d. What the project team and discontinued learners said 

• Recruitment and retention 

a. What learners said 

b. What education staff (both paid and volunteer) said 

c. What managers/supervisors said 

d. What the project team and discontinued learners said 

• Staff development 

a. What learners said 

b. What education staff (both paid and volunteer) said 

c. What managers/supervisors said 

d. What the project team and discontinued learners said 
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Analyzing the data 



The coordinator now had in front of her a pile of pages filled 
with responses. To help the team to analyze those responses, she 
first read through them and began jotting down her own personal 
observations, questions, ideas, and interpretations. She recorded 
these under the heading of "coordinator's interpretations." 

She then sent to each team member a summary of the data 
gathered along with her personal interpretations. She asked each 
team member to essentially do what she just did: review the 
responses and jot down his/her personal interpretations. 

In its third meeting (on August 6th), the team shared their 
interpretations, keeping in mind the following questions prepared by 
the facilitators: 

Qjiestions to Consider When Interpreting Data 

• What conclusions can we draw from these responses? (How 
should we interpret these responses?" What reactions do you 
have?) 

• Do certain points need clarificatj.on? 

• Do you agree or disagree with, particular points? 

• Are there any surprises here? 

• Are there any conflicts or contradictions in the data? 

• Which recommendations should we "keep" in our final report? 

• Which recommendations need to be changed? 

• Which should we not include in our final report? 

In response to each section of the data, team members 
brainstormed various observations, questions, and reconmiendations 
which were recorded on flipcharts. 

Preparing a strategy for reporting the findings 

To pull these findings together in one or more final reports for 
its intended audiences, the team agreed on the potential formats for 
each audience, who would be responsible for preparing draft reports, 
how and when they would be reviewed by the team, how and when 
the report(s) would be presented, and how the team should be 
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prepared to take follow-up actions after the reports were presented. 
They prepared the following timeline, tying these elements together: 

Preparing the VGH Report : 
Things to Do 

1. Audience : Aim at the President's Executive Council (PEC) in 
particular. 

2. Formats : Prepare both an executive summary and a more- 
detailed version. Be sure to include all responses given. 

3. Who responsible : Christine (the coordinator) is in charge of 
preparing the report, with help from others (including possible 
editing help from ABC CANADA). Explore getting another 
computer to help with editing. 

4. Timeline for preparation of report: 

• Draft due : approximately September 6 

• Final report due : mid-September. 

5. Presentation to PEC: approximately late September. (Training 
and Development representative will arrange this presentation.) 

With that mandate, the coordinator prepared a final report, 
presenting each section of data and the team's recommendations for 
various concrete actions related to those findings. Also included in 
the detailed report was a brief history of the program, a history of 
the evaluation project, a description of the evaluation methodology, 
and appendices with details of the instruments and information 
sources used. An executive summary was included, summarizing 
key findings and the team's recommendations. (See Appendix.) 

Reporting the findings 

hi late September, the coordinator circulated copies of the 
report to the team. They agreed that the team member with closest 
links to the team's primary audience, the President's Executive 
Council, would arrange to have the report presented to the Council 
and other intended audiences. 

One of the first audiences to see the report was the new 
instructor who was replacing the outgoing project coordinator. She 
found that it helped her understand the program's history and needs. 

hi early December, the head of the Training & Development 
Department responded very favorably to the report. (See references 
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to "the most senior management representative" under "Evaluating 
the evaluation" below.) He constituted the team's link to the 
President's Executive Council and had the authority to see that many 
of the report's recommendations were implemented. 

Phase IV : Deciding what happens next 

Taking follow-up action 

To anticipate what evaluation-related activities might follow 
the presentation of the report, the team ~ at the August 6th meeting 
-- reviewed its action plan of April 19th. Now that they were 
ending the first round of data-gathering activities, the team agreed 
to continue incorporating evaluation into its planning activities, 
emphasize the generation of quantitative data, and clarify what the 
new instructor should do regarding evaluation. 

In reality, the team has taken the following actions since the 
report was presented to the various audiences: 

• The program has been refunded at higher levels. 

• A new coordinator was hired, with more hours thai^. the 
previous coordinator had. 

• Classes were resumed in the fall with 32 learners and 11 tutors 
(up from 20 learners and 10 tutors from the previous year)> VGH 
continued to increase funding as additional learners signed up. 

• VGH allocated permanent secretarial help, office space, and a 
classroom. 

• The coordinator had, by early December, begun planning mini- 
workshops for learners as well as a student handbook. 

• The instructor tried using tutors as aides in the math class. This 
idea, however, didn't work out as hoped and was discontinued. 

• Three former participants are now serving as tutors, something 
not done before. Their department has given 50/50 release time 
to allow the participants to do this. (Their department head is a 
former tutor and team member herself, and has consistently 
championed the program.) 

Due to major changes going on elsewhere in the hospital, by 
December several key team members were no longer able to attend 
team meetings. Those remaining on the team agreed that they 
needed "new faces" on the team, as "reinforcements." 



17 348 



Those who remained on the team also agreed that evaluation 
needed to be continued in some way -- even if not as extensively as 
had been hoped for in the original evaluation strategy -- because "it 
is too good to let go of completely." One option considered: reduce 
the number of data-gathering activities and concentrate on a team 
meeting in which members prioritize actions they need to take. 



Evaluating the evaluation 

In the August 6th meeting, the team assessed the evaluation 
process, as follows: 

Team Members' Feedback 
About the Team Evaluation Process 

Goals/expertari ons for the evahiation project : 

VGH's expectatarions ! 

• To learn how to evaluate the program. 

• To develop a process of evaluation. 

• To determine outcomes and needed improvements for the 
program. 

• To get information needed to justify funding. 

• To document the value of the program. 

ABC's/U.S. Nati onal Institute for I.iteracv project's expectations : 

• To develop a process of evaluation which others might adapt. 

What was actually accomplished : 

• All of VGK's expectations were achieved except developing 
more-specific, clear evidence of program outcomes, especially 
work-related ones. (But the team doesn't want a standardized 
test.) 

• The team has in general now developed a foundation for 
further evaluation work. 

• Timing of the project was good, as the team wanted to do an 
evaluation at just the time that the facilitators were looking 
for a site to work with. 

Strengths and limitations of the project : 
Strengths: 
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• The facilitators' help was fast, responsive. 

• Their faciiitative style "brought it out of us," helped team 
members do it themselves. 

• VGH provided good support. 

• Good documentation was kept of the project. 

• Project built team identity, brought out members' strengths. 
Limitations : 

• Project took a fair amount of time. 

• The facilitators were far away. 

• Members didn't have a com.piementary computer and modem 
for easy communication, editing, etc. 

• Data-analysis step could have been abbreviated with clearer 
guidelines. 

How the process might be improved in the future : 

• Give members training on data-collection before asking them to 
conduct interviews, etc. 

• Get additional feedback from people who did data-gathering 
about how it went. 

• Revise data-gathering tools used as soon as possible, so we don't 
forget problems we ran into or effective ways of using tliem. 

• Pilot the tools and revise them before fully using them. 

• Consider following up individually with members of focus 
groups, if it appears they might have more to say in private. 

• Ensure larger/broader representation of supervisors when they 
are interviewed. 

• Be careful not to be too ambitious in an evaluation activity (e.g., 
trying to gather too much information from too many sources.) 

• Get learners to be more concrete and identify their goals on an 
ongoing basis. 

Looking back on the process several months later, the original 
project coordinator raised several issues: 

• There is a need to avoid using evaluation as a way of judging 
learners. Maintaining confidentiality is vital, to avoid threatening 
learners in any way by revealing information about their basic 
skills to others who might use it to manipulate learners. 

• The facilitator needs to be careful not to over-influence team 
members' interpretations of the data. At the same time, there is a 
value in the facilitator sharing his/her interpretations, to provoke 
further thinking by other team members. 
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• The presence of "outside experts" as facilitators gave a certain 
legitimacy to the process, providing a kind of "checks and 
balances" to reduce the possibility that the internal team would 
allow their biases to shape the findings. The outsiders also were 
able to guide inexperienced team through the steps, helping to 
make the process as smooth as possible. 

The coordinator also felt that the evaluation process had not 
only enabled her to contribute to the program but given her an 
opportunity to develop new professional skills. She called it "a 
fantastic experience," a high-point in her career. 

Another team member felt that the evaluation helped create a 
solid foundation for the program, particularly by establishing good 
relations among stakeholders. This ~ coupled with the good word 
put out by satisfied learners -- led to increased attendance. She also 
felt that the report showed that tlie program was "accountable." She 
will include a summary of the report's recommendations (especially 
for a full-time instructor) when she submits the next funding 
proposals. 

The most senior management member on the team was 
reportedly very pleased by the report and the evaluation process. 
He felt it was done professionally, and he felt its use of a number of 
informants gave the process validity. He felt it provided a road map 
to follow, a shared vision for the program's future. This was 
particularly important to him, as he was relatively new to the 
education program. 

He was also pleased that the recommendations were realistic 
and that team members took the initiative to begin cairying out 
many of them, without waiting for him to locate resources and figure 
out how to respond to the report. This showed the potential of the 
self-directed work teams being implemented in the hospital. 

Two months after completing the report, the team continued to 
refer to its recommendations, using them as a guide to see "how are 
we doing." However, some of the key players on the team have now 
left, due to changes in personnel unrelated to the evaluation process. 
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rv. REFLECTIONS 



Facilitator Paul Jurmo's comments: 

The VGH team was successful in producing a useful evaluation 
for several interrelated reasons: 

• Team mem.bers were motivated to do an evaluation from the 
start, having requested help. They thus saw a need and saw 

. the value in committing time into tlie process. This allowed 
them to be flexible and patient, ^villing to work with the 
facihtators to adapt the evaluation process to their own 
situation. 

• Several team members had considerable expertise in adult 
education, evaluation, or other relevant areas. Those lacking 
such experience v^ere nonetheless open - and felt safe enough 
~ to stretch themselves and take on new roles in the 
evaluation process. 

• The hospital gave members the time and other resources 
they needed to participate in the process. 

• The education program had already created a climate of good 
will among team members. This contributed to a sense of trust 
and an interest in working together further on the evaluation. 

• They had an audience who had a demonstrated interest in 
the program, maidng it worthwhile for the team to go through 
the effort of collecting considerable information, analyzing it, 
and reporting it. 

• The team had a good mix of perspectives, including one 
program participant. All team members seemed willing and 
able to participate actively. 

The process did not produce the kind of quantitative data on 
program impact which the team felt its primary audience would 
like. Instead, the evaluation produced less-easily-quantifiable 
data on impact and focused more on formative feedback on how 
to improve tiie program. 

The process did take time (three day-long team workshops 
plus preparation, data-gathering, report-writing, etc. by 
individuals). This investment of time, however, seemed to 
produce useful results. 
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The evaluation produced useful information about the possible 
formats and content of a workplace education evaluation. It also 
produced useful information and analysis about one particular 
program, something rare in the workplace education literature. 
Team members also developed expertise in various facets of 
evaluation work, and demonstrated the potential of shared 
decision-making in tlie workplace. 
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EXECUTIVE SUMMARY 



INTRODUCTION 



The Workplace Education Project Team of the Victoria General 
Hospital, in consultation with ABC Canada and Literacy 
Partnerships, Jersey City, planned and conducted an evaluation 
of the Workplace Education Program. 

This process necessitated that members of the project team 
acquire an in-depth knowledge of the program, and become 
versed in evaluation skills. These individuals exemplified the 
team spirit as they worked towards their goal. 

Learners and instructors past and current, volunteer tutors, 
management/supervisory staff and the project team were 
interviewed for this report; a total of thirty-eight 
individuals . 



The evaluation serves a number of purposes. It provides 
information about the principles and practices of workplace 
education, examines goals and achievements to date, and offers 
suggestions for planning future programs. Such documentation 
provides information of potential use in funding decisions. 
The team made the following conclusions based on their 
analysis of the data. 



CONCLUSIONS 



Conclusions are discussed in order of the four major questions 
asked in this evaluation. 



What have been the goals of the program? 

Most participants have a clear understanding of program goals. 
Learners, education staff, some supervisors and project team 
understand the range of needs and interests being met by this 
program. 

Some supervisors need a clearer idea about workplace 
education, what its purposes are, and how they can support 
learners in the area of release time as well as giving them 
more challenges on the job. 



Learners are better able to work 
attending workplace education 
informed about the program. 



out suitable arrangements for 
after supervisors are well 
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what has been accomplished to date? 

The program has been successful in achieving its goals. It has 
also achieved unanticipated benefits for individual learners, 
others who participated in the program, and the organization. 

Some unanticipated benefits include increased morale , self- 
conf idence and greater participation in departmental 
activities. In addition, the program and its evaluation is now 
a model for other workplaces in Canada and the United States. 

Hospital employees gained on the job^ skills and self- 
confidence as a result of tutoring in the workplace education 
program. 

There is confusion about confidentiality. Supervisors need to 
be clear about this issue. "We are in a Catch 22 situation; 
maintaining confidentiality versus getting support from 
supervisors. Supervisors can't help participants if they don't 
know who the participants are". 

There is a need to define each of the two areas in v/hich 
"confidentiality" can be an issue. 

1. knowledge of employee participation 

2. employee goals and progress 

Should the program be continued and why? 

Learners, education staff, supervisors and project team want 
the program to continue for work-related as well as personal 
reasons . 

What actions need to be taken to improve and expand the 
program? 

The on-site location is convenient and accessible for 
employees; however, a permanent class and office space are 
required. 

The no-cost policy increases access for employees. 

Employees like the option of working with tutors as well as 
the "learner-centred" approach which encourages input from 
learners into content, materials, homework and so forth. 

Instructors can feel a sense of professional isolation. They 
need support within the Victoria General Hospital and also 
from colleagues in other organizations. 
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Scheduling of classes is important. The time of day and/or 
other priorities can cause some employees to discontinue the 
program. 

The team made the following recommendations based on the above 
conclusions. 

Ill . RECOMMENDATIONS 

A. Introduction 

The following short-term recommendations should be implemented 
prior to starting the '93-'94 workplace education program. 

The long-term recommendations will take time to develop and 
implement. However, the project team, coordinator and 
instructor should begin work on them immediately. An overall 
plan which develops goals and a time-line for implementing 
these long-term recommendations should be established with the 
first or second meeting of the project team in the ' 93 - ' 94 
workplace education term. 

Those recommendations which involve hospital personnel other 
than workplace education staff and project team are 
suggestions put forth for their consideration. 

B. Short-term Recommendations 

Classes are to continue to be held on-site but in a location 
that does not require moving the class from room to room. 

Option A: Get a large room for classes that have a phone 
and space for administrative work and filing 

Option B: Provide office space and a separate classroom. 
Both are to be dedicated to workplace education only. 

Continue : 

to offer a tutor training course and have volunteers work 
one-on-one with learners. 

to build a library of materials 

to have the coordinator, instructor and 
to assist learners and supervisors 
arrangements and any other needs . 

Increase instructor's hours for one-on-one 
counselling and extra academic assistance. 
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tutors available 
with scheduling 

program-related 



Offer greater flexibility in terms of class times to 
accommodate shift and personal demands. Potential learners are 
to continue to have input into most suitable class times. 

Provide the option of more time for each class/ learning 
session- Each class negotiates how much time it needs/has 
available . 

Encourage tutors to help in group math classes as the 
instructor sees the need . 

Give supervisors a minimum of two week's notice of class start 
date and schedules. 

Improve communication and increase information to supervisors 
about program goals and purposes. 

Supervisors should be kept informed about the program in 
general, whether or not they know identities of individual 
participants . 

The coordinator and instructor should expand the current 
practices for recruitment, orientation and retention. 
Incorporate suggestions set forth in this report. 

Long-Term Recomiuendations 

Develop clear guidelines about confidentiality. 

Option A: Supervisors know identity of learners, who are able 
to attend partly on hospital (release) time. 

Option B: Supervisors do not know identity of learners v;ho 
then participate on their own time. 

Supervisors should maintain confidentiality but continue to 
support employees . 

The project team should propose a 50/50 release time policy, 
with the hospital's share being either release time or 
equivalent. But consider ceiling for the maximum number of 
hours an employee attends classes. 

Develop a handbook for all levels of employees which outlines 
the purpose and practice of workplace education. 

Develop av/areness workshops for supervisors. Within these 
activities, clarify how supervisors should respond if an 
employee approaches them with a basic skill problem. 

Involve supervisors experienced with workplace education as 
co-facilitators in awareness sessions. 
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Expand the topics and have more mini-courses. Memo-writing, 
telephone skills, public speaking and report writing are some 
suggestions. These short sessions will decrease schedule re- 
adjustments on the work-site and reach more people with 
special needs and interests. 

Include purchase of a computer in selection of equipment and 
materials. 

Investigate resources to assist learners with information 
about continuing education, start with Nova Scotia Community 
College (NSCC) and Employee Assistance Program (CARE) and 
unions. 

Investigate other funding partners. Funding may be sought for 
specific activities in contrast to overall program 
development . 

Continue to search for sources (time and funding) for 
professional development. 

Workplace education instructor should be full-time staff 
member with the Training and Development Department. 

The status of Workplace Education is to be raised to that of 
other training programs in the hospital. 

Summary statement 

The evidence shows that overall, the Workplace Education 
Program has worthwhile goals. The achievement of such goals 
has benefitted individuals and the organization. In addition, 
unanticipated but welcome outcomes have resulted! 
Implementation of the short and long term recommendations will 
ensure that not only will quality programming continue, but 
some areas of the program will be improved and expanded. 

A special characteristic of this evaluation is that the team 
has been involved in the entire process. They planned the 
strategy, conducted the interviews and analyzed the data for 
conclusions and recommendations. This unique approach ensures 
that the evaluation questions: "...stem from the information 
needs of relevant and identified decision makers and 
information users," (Patton, 1978,' p. 201). Furthermore, the 
skills gained by the employees involved in this process remain 
with the organization for application in other areas. 
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The project team is to be given credit for their hard v;ork and 
dedication to a program which has touched the lives of many 
and made significant changes in the lives of others over the 
past three years. 

The Victoria General Hospital is to be commended for providing 
such a program. Since implementation of this evaluation, 
increased funding has been approved for 1993 - 1994. 
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DEFINITIONS 

Supervisors /Managers : used throughout this report to include 

department heads , managers , supervisors 

Learners: Employees who participated in a tutoring 

relationship or classes 



Release time/ 
Time share/ 

Matched time: Arrangement an employee makes with 

employer to have part of time spent in 
classes or with tutors paid by employer or 
the time spent on shift is reduced by a 
percentage of time employers is in classes 
or with tutor 



INTRODUCTION 



Overview of the Evaluation 

At the January 1993 meeting of the Project Team, Christine 
Garlick-Grif f in, Instructor, gave a presenilation on the value 
of the team approach to formally evaluating the Workplace 
Education Program. The team agreed to proceed, and Sue 
Folinsbee, workplace consultant to ABC Canada and Paul Jurmo, 
researcher. Literacy Partnerships, Jersey City, United States, 
provided consultation. They are incorporating elements of this 
evaluation into workplace education research in Canada and the 
United States. 

In addition to ongoing long distance communication, the team 
held three days of workshops with Sue and Paul at the Victoria 
General Hospital (VGH) . An evaluation action plan was 
established and finally the data analyzed for conclusions and 
recommendations. The process and results are documented in the 
following report. 

Orcranization of the Report 

Section two describes the V^/orkplace Education Program. Section 
three discusses methodology. Section four presents the 
findings. Section five lists recommendations. 

WORKPLACE EDUCATION PROGRAM AT THE VICTORIA GENERAL HOSPITAL 
Procrram History 

In the fall of 1990, Angle Gillis, then consultant for 
Training and Development at the hospital, attended an 
information session on workplace education programs at what is 
now the Literacy Division of the Nova Scotia Department of 
Education. Angle recognized the value of such a program. She 
presented her ideas and the hospital agreed to the 
implementation of a Workplace Education Program. 

Six employees from Food and Nutrition Services registered for 
this first program in 1990 - 1991. The following year the 
program was expanded to include housekeeping and the laundry 
departments with approximately 25 registrants. The Department 
of Education delivered the program both years. They supplied 
one instructor in 1990 - 1991 and two in 1991 - 1992 program. 

In 1992 - 1993, the hospital funded a part-time person to plan 
the program, instruct classes and coordinate volunteer tutors. 
The Department of Education continued their support. They 
provided a thirty hour training session for 10 volunteers, 
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including eight hospital employees, to become tutors, offered 
ongoing professional development for the workplace person, 
maintained two seats on the project team and provided 
consulting services on an as-needed basis. The 1992 -1993 
program expanded to eight departments and registered 20 
learners . 

At present, the program is situated with Training and 
Development and coordinated by Angle Gillis who is presently 
with Food and Nutrition Services. The Workplace Education 
Program began in 1990 and at the writing of this report is 
preparing for its fourth year. 

B. Program Goals 

The goals of the program are: 

"To enable participants to develop self-esteem, 
reading, writing, oral English and math skills 
needed to solve problems they identify as of 
interest to themselves". 

In other words, the program has a learner centered approach. 
Some learners may choose to learn a specific skill such as 
applying percentages in the workplace or memo and report 
writing. Others may study for the Grade Twelve Equivalency 
Exam (GED) . 

C. Procrram Philosophy 

In contrast to the more traditional "school" approach to 
education, the program incorporates adult learning/teaching 
principles. Learners define their needs and interests and take 
responsibility for their learning. Furthermore, this approach 
encourages active participation in program planning and 
curriculum development. 

D« Procrram Principles 

An empowerment model of workplace education has been 
implemented. This model assumes employees have highly 
developed skills in some areas but may want to upgrade basic 
skills. Other principles of workplace education practiced 
include: 

• voluntary participation 

• confidentiality of individual's progress 
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Continuous Quality Improvement 

The content, philosophy and principles of the Victoria General 
Hospital Workplace Education Program reflect the values of 
Continuous Quality Improvement (CQI) . The program provides an 
opportunity for frontline employees to improve computation , 
reading, writing and problem solving skills. Such skills are 
a pre-requisite for achieving those aspects of CQI which 
encourage increased morale, teamwork and better overall 
performance in the v/orkplace. 

Program Components 

The program consists of the project team, small group learning 
or classes and volunteer tutors . 

The project team is the liaison between the worksite and the 
program and advises the instructor on workplace issues . It 
meets on a monthly basis or more often in the case of special 
projects such as the evaluation. The project team has 
representatives from management, union. Department of 
Education, tutors and learners and is coordinated by the 
workplace instructor . 

The classes are small groups of employees which usually number 
no more than twelve. They are organized so that the 
instructor/ learner interaction is a mixture of one-to-or3 and 
small group instruction. 

Learners who are either unable to attend classes or prefer 
one-to-one learning situations are matched with one of the 
specially trained tutors.. Tutors and learners take 
responsibility for organizing what they will learn, when and 
where they meet and prepare a report for the instructor 
monthly. In addition, tutors and instructor meet on a regular 
basis to share ideas and concerns. 



G# Physical Setting 

All classes are held at the hospital in the Bethune Building. 
Rooms are booked for the two hour sessions. There is no 
consistency in space allocation from class to class. Materials 
and flipcharts are carried to and from each class. The 1992 - 
1993 program was fortunate in the latter part of its term. 
Marlene, in roombooking, made special arrangements with Allied 
Health Services for temporary loan of a teaching room in 
Bethune. Temporary office space, first with Training and 
Development, then on the fifth floor in the Bethune Building 
was also provided . 
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Potential Learners 



The Program is open to any hospital employee who chooses to 
learn or refresh basic skills. To date, learners have included 
employees from Food and Nutrition, Modern Cleaners, NovaCos 
Cleaners, Laundry, Central Escort Services, Bio-Medical and 
three nursing units. Employees in supervisory positions also 
register in this program. 



In summary, the history, description and rationale for 
workplace education have been discussed. In the following 
section , the evaluation methodology is examined . 

III. METHODOLOGY 

A. Introduction 



In this section of the report, purposes for evaluating the 
program are discussed, reasons for using this particular model 
are explained, the major steps in the process are described 
and finally, the advantages and limitations of this approach 
are presented. 

B. Purposes of the Evaluation 

This evaluation serves a number of purposes: 1) assists 
with funding decisions; 2) provides information about the 
purposes and practice of workplace education; and 3) documents 
program accomplishments as well as suggestions for improving 
or expanding the program. 



C. Why this type of Evaluation 



The Workplace Education Program at the Victoria General 
Hospital does not lend itself to being judged on a single 
criterion of academic achievement. The broader approach, which 
includes analyzing program context, is in keeping with adult 
education principles and is more effective in demonstrating 
the worth of the program . 

Furthermore, the team approach referred to in this document 
is quite different from a traditional evaluation in which one 
or more trained individuals work together only for the 
purposes of carrying out an evaluation. The team is made up of 
mostly hospital employees who represent various interest 
groups in the hospital. These team members are dedicated to 
ensuring good quality programming for the employees. 
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D. 



Steps in the Evaluation Process 



The team decided to proceed with this evaluation on January 
27. They met eight times over a seven month period. The final 
workshop was held on August 6, 1993. 

i. Preparation Activities 

Initial meetings of the evaluation team focused on group 
discussion and analysis of the concept of evaluation and its 
application to the Victoria General Hospital Workplace 
Education Program. 

The team stated; 

"OVERALL: Through discussion and research. . .we can decide 
what we want to know, why and how to start getting that 
information. We need to rework guesswork so we can 
instead develop a simple and useful evaluation strategy. 
(That is, provide us with a clear, concrete, and 
unequivocal direction for our evaluation activities)." 

Following this preparatory phase, the team proceeded through 
the following steps: 

• decided what kinds of information they wanted to know 

• decided who they would gather this information from 

• prepared the appropriate instruments 

• gathered the data 

• organized, analyzed and interpreted the data 

• prepared the final report 

ii* What did the team want to know? 

The team wanted answers to the following questions: 

• What have been the goals of the program? 

• What has been accomplished to date? 

• Should program be continued and why? 

• What do we need to do to improve and/or expand the 
program? 

In order to get this information, each of the above was 
further divided into questions which would licit more 
detailed responses. (See appendices) 

iii. Who were the sources of this information? 



• past program participants 

• current program participants 

• Management/supervisory staff who had employocs in the 
program 
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• instructors and tutors (past and current) 

• individuals who came for initial interview but did not 
register and those who started the program but 
discontinued (referred to as discontinued 

learners /participants) 

• project team 

Although forty-three (43) responses make up the data for this 
evaluation, it was provided by thirty-eight (38) individuals. 
This overlap occurs because five (5) of the project team were 
also interviewed in their other roles. 

No particular selection process dictated who in each of these 
groups would be interviewed. Availability and willingness to 
be interviewed were the only criteria. 

iv. Col lection Methods 

Individual face -to -face interviews , focus groups as well as 
questionnaires and one telephone interview were used to gather 
information. From a discussion group on March 29, Sue and Paul 
recorded responses form eight project team members. Individual 
interviews were conducted with eight 1991 - 1992 learners, 
three 1992 - 1993 tutored learners, three discontinued 
learners and one tutor. One discontinued learner was 
interviewed by phone. Six supervisors were interviewed in a 
focus group, one supervisor who could not attend the focus 
group provided a written response. Eight 1992 - 1993 
learners were interviewed in a focus group. Three instructors 
and one tutor provided written responses to the questions. 

Objectivity was maintained by asking the same questions to all 
members of the same group. Most questions were open-ended. 

V. Arranging the Interviews 

Six members of the evaluation team and a consultant from 
Training and Development conducted the interviews and focus 
groups. Interviewers and interviewees were matched to get 
maximum objectivity. For instance, an instructor did not 
interview his/her learner. All interviewees were promised 
confidentiality. Individual interview times ranged from 
approximately one-half to one hour. Focus groups took one and 
one-half hours. Interviews took place at various locations 
within the hospital including the classroom and cafeteria. In 
some instances, there was difficulty making contact with 
potential interviewees as well as finding mutually convenient 
times for interviews. As a result, two (2) learners, two (2) 
discontinued learners and one (1) supervisor did not get 
interviewed as originally planned. 



6 

370 



vi • 
a. 



Advantages and limitation of this approach to Gvaluatlon 



Advantages 



This evaluation examines perceptions and opinions of those 
involved in workplace education. Cyril Houle (1972), well 
known adult educator and researcher, states: 

"The analysis or planning of educational activities 
must be based on the realities of human 
experience..." (p. 32). 

The consultants who facilitated the process were outside the 
organization and had no stake in the results. Furthermore, the 
team approach ensured all interest groups were represented as 
well as maintained a degree of objectivity in the process. 

Validity of the results is demonstrated by the many 
similarities in responses from the different sources. 
Furthermore, the similarities remained consistent whether the 
data was gathered by face-to-face interview, questionnaire or 
focus group. 

The members of the evaluation team have new skills which can 
be used in future evaluations of the Workplace Education 
Program as well as in other aspects of their work. 

Limitations 

It is recognized that in a qualitative evaluation some 
subjectivity exists. 

As indicated, a variety of methods were used to collect data 
including focus groups and interviews. All methods have 
advantages and disadvantages. For instance, in focus groups, 
some individuals may be reluctant to express opinions which 
they feel do not meet with favour from the rest of the group. 
This situation is more likely when supervisors and management 
from varying levels in the organization are in the same focus 
group. AlsO; some learners may feel intimidated in face-to- 
face interviews. This could result in limited responses to 
some questions. 

In summary, this section on methodology explained the purpose, 
rationale and the steps in the evaluation process. Also, the 
advantages and limitations of such process are explained. In 
the following section, findings are discussed. 
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IV. FINDINGS 

A. Introduction 

The findings are presented in four sections. In each of the 
first three, summaries of the responses are organized 
according to the sources of information- The fourth is 
organized according to types of information. Conclusions are 
found at the end of each section. The sections are: 

• What have been the goals of the program? 

• What has been accomplished to date? 

• Should the program be continued and why? 

• What actions need to be taken to improve and expand the 
program? 

B. What have been the goals of the procrram? 

Goals of the program have been identified by the groups 
interviewed as increasing basic skills and self-confidence for 
work-related, academic and personal reasons. 

i . Learners 

Fifty-three percent (53%) of the learners interviewed stated 
they wanted credit for Grade Twelve Equivalency (GED) . Others 
wanted to increase their skills in math, writing and reading. 
Interest in improving memory, learning ability, study skills 
and self-esteem were also expressed. 

Learner Comments 

• "Main goal - to receive a high school diploma." 

• "I wanted to learn more writing and math. Before I 
started, I only knew how to write my own name. ..I don't 
feel as bad about school as I used to." 

• "Being relaxed. Learn about your self-esteem." 
ii • Learners who Discontinued Program 

Three of the four interviewed stated they had hoped to get 
their GED. Helping children with school work, learning math 
and to public speak were also stated. 

Discontinued Learner Comments 

• "I had hoped to get a better, understanding of math. I 
got a lot of confidence when I was in the program ...I 
would have liked to do more public speaking." 

• "Grade XII and learning a little bit more." 
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iii. Education Staff 



The education staff observed that learner's work-related goals 
included responding to department memos, filling out forms and 
preparing for other jobs and promotions. Learners wanted GEO 
in order to pursue other education and for self-satisfaction. 
Personal goals such as budgeting, using cheque book and 
helping children were noted. The 1992 - 1993 class wanted GED, 
math, communication (reading and writing) , self-esteem and 
teamwork for personal, community and work-related application. 
Staff noted that most common goals were math, oral 
communication and writing. 

Education Staff Comments 

• "Main goals of my learner- improve writing and math 
skills to feel more confident in filling out forms and 
responding to department memos," 

• "They wanted the ability to write effectively and 
communicate orally.. .at some point to write GED but 
many wanted to think about other jobs and promotions." 

• "Others wanted to improve their writing, reading or 
math skills for their own satisfaction, to help their 
children or go on to further education." 

iv. Managers /Supervisors 

Two supervisors stated they had no particular expectations of 
their employees . Once said "no pre-conditions" placed on 
employee. Others hoped for increased self-confidence, improved 
problem-solving and communication skills. 

Some of the six supervisors were unclear about individual and 
program goals. One said more was learned as the program 
progressed and another stated: "should know more belore ouv 
employees become involved. " 

V. Project Team 

The eight members of the project team identified a number of 
goals, many for work-related purposes. They talked of the need 
for math and writing skills to check information on job- 
related requirements and procedures, to read recipes nnd menus 
in food services area, and to understand "universal 
precautions" to speak up with ideas and get to knov; other 
employees. They also saw the program as an opportunity for 
employees to assist children with homework, get their (jED and 
fee 1 more respected by others . 
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vi ♦ Summary 



Learners have many reasons for participating in workplace 
education* They want to perform better on the job, to pass the 
GED and to function * better in their personal lives. Learners, 
education staff, the project team and some supervisors 
understand that the program serves a specific academic need as 
well as providing practical and useful skills. Some 
supervisors are unclear about program purposes . 

vii* Conclusions 

Most participants have a clear understanding of program goals. 
Learners, education staff, some supervisors and project team 
understand the range of neads and interests being met by the 
program . 

Some supervisors need a clearer idea about workplace 
education, what its purposes are, and how they can support 
learners in the areas of release time as well as giving them 
more challenges on the job. 

Learners are better able to work out suitable arrangements for 
attending workplace education after supervisors are well 
informed about the program. 

C» What has been accomplished to date 

Skills and confidence to perform better in the workplace, GED 
credit and/or increased academic achievement as well as 
practical skills to assist in the home and community are 
results of the workplace education program. 

i. Learners 

Sixteen (16) of nineteen (19) learners stated they were 70%- 
100% successful in achieving their goals. Others stated they 
were in the process of achieving their goals but more time was 
needed. The 1992 - 1993 learners wanted it to be clear that 
their reasons for not fully meeting goals were not related to 
the course but were for other reasons such as illness or not 
yet having written their GED exam. 

Eleven (11) learners mentioned self-confidence and eight (8) 
of those stressed its importance.- In addition, examples of 
increased self-confidence were cited. One supervisor is no 
longer afraid when training new employees. Better leadership 
ability, more confident at trying new. things, no longer afraid 
to speak out and decreased shyness were also noted. The 1992 - 
1993 learners emphasized that they are now more aware of their 
knowledge and abilities. 
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other examples of what was accomplished: math, reading, 
writing, oral communication, expression, social skills 
(meeting and learning about other employees) , creativity, time 
management, accepting criticism, making a commitment, helping 
kids with school work, asking questions, expressing opinions 
and teamwork ♦ 

Learner Comments 

• "Can train without fear*" 

• "Not afraid to give opinion — couldn't before." 

• "How to manage time between work, family and 
education. " 

• "I understand what I read now which helps me to relate 
to my daughter." 

• "Learned to write a story. Had never written a story 
before in my life." 

Education Staff 

All staff noted that learners achieved a wide range of 
behaviours and attitudes. In addition, they estimated 
learner's success as 70%-100% in achieving their intended 
goals. Frequently stated accomplishments include: increased 
social skills such as self-confidence/worth, interpersonal 
skills and teamwork. Problem-solving skills such as handling 
tensions at work and increases in academic learning are also 
noted. One instructor said that not only did learners 
discover that their input and participation were welcome and 
valued in the Workplace Education Program, but these skills 
transferred directly to increased input at work. 

Education staff also noted unexpected benefits resulting frc:n 
the program. One tutor became much more aware of importance of 
readability of hospital materials and sensitized to the 
teaching and counselling methods she used in her work as an 
hospital employee. Another found the tutoring particularly 
challenging but gained personally because it resulted in a 
positive experience for both tutor and learner. The program 
also resulted in the Victoria General Hospital having 
increased visibility as a workplace education model nationally 
'and internationally . 

Education Staff Comments 

• "They valued that their input and participation is 

welcome and valued which translates directly to 
increased input at work." 
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• "They learned how to solve problems and think 
independently. They gained teamwork skills. Some even 
applied for other programs. As a result, they recognize 
the importance of training at work." 

• "To analyze one's problems, verbalize a solution and 
act on it." 

• "To surmount workplace and personal barriers to 
learning. " 

iii • Managers /Supervisors 

Fifty percent (50%) of the six (6) managers/supervisors 
interviewed observed an increase in learner/employee self- 
confidence. One of the six noted a decrease in one employee. 
Others noted more assertive behaviour in the workplace and 
more involvement in staff meetings and other department 
activities. One supervisor noted that a learner who was 
formerly secretive about attending the program became more 
open about it. Another observed learners talking about the 
benefits of the program to other employees. Furthermore, other 
workers are now interested in joining the program and in most 
cases, other workers are supportive of co-worker involvement. 

Some of the six supervisors interviewed had concerns. For 
instance, one noted some stress levels rose due to increased 
workloads and schedule changes. One found it time consuming to 
decide about release time. "Has had to search ... soul . Have 
been unable to allow her to attend on work time — in order to 
be fair to other staff who might request time off for 
education...". Another supervisor had a question: "Is it fair 
to one staff member if they cannot attend on work time, if 
those in other departments can?" Confidentiality was also an 
issue. One ^supervisor said: "The desire for confidentiality 
may leave out supervisor who may need to be involved in 
timing. " 

iv* Project Team 

The project team described a wide range of program strengths. 
They observed that the program fills a real need, has support 
from all levels and the organization doesn't place undue 
demands/stress on the program and' learners . Not only regular 
workers, but supervisors also take classes. One member states: 
"It is the most significant staff development opportunity at 
the VG." t^i- 

They noted that the 1992 - 1993 program had eight (8) 
departments involved. Many departments have established a paid 
release policy, developed flexibility with scheduling and 
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respect that learner's skill levels and participation are 
confidential . 

They state that the program is flexible and responsive to 
individual needs, "employees don't get lost in unresponsive 
groups • " Tutors are mentioned specifically for their 
flexibility and responsiveness • The on-site location and time 
of program is also convenient. 

As a result of the program, learners develop self-esteem and 
motivation. Their participation is also supported by friends 
and co-workers. Furthermore, other departments, such as print 
shop, audio visual. Training and Development and Food and 
Nutrition Services offer concrete support. 

Summary 

Learners, education staff, supervisors and project team talked 
about a wide range of behaviours and attitudes achieved as a 
result of workplace education. They observed that not only 
individual learners but other program staff and the 
organization benef it ted . 

Overall, learners were successful completing their goals 
within the twenty-four weeks. In addition to improving basic 
skills, many experienced increased self-confidence on the job 
and in their daily lives. 

Some supervisors expressed concerns about stress, arranging 
schedules, workloads, release time and the issue of learner 
confidentiality . 

Unexpected benefits include volunteer tutors applying 
knowledge and skills learned in workplace education to their 
work as hospital employees dealing with patients. Also, the 
program now serves as a model in Canada and the United States, 

Conclusions 

• The program has been successful in achieving its goals. 
It has also achieved unanticipated benefits for 
individual learners , others who part ic i pa ted in the 
program, and the organization. 

• Some unanticipated benefits include increased morale, 
self-conf idence and greater participation in 
departmenta 1 activities . 

• Hospital employees gained on-the-job skills as a result 
of tutoring in the Workplace Education Program. 
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• There is confusion about confidentiality. Supervisors 
need to be clear about this issue. "We are in a catch 
2 2 situation; maintaining conf identiality versus 
getting support from supervisors . Supervisors can' t 
help participants if they don't know who the 
participants are. " 

• There is a need to def ine each of the two areas in 
which "confidentiality" can be an issue: 



1 . knowledge of employee participation 

2 . employee goals and progress 

D. Should the program be continued and why 

All individuals stated the program should be continued. 
i# Learners 



All learners interviewed stated the program should continue. 
The 1992 - 1993 learners want it known that they felt strongly 
about the program continuing. 

Learners stated a number of work-related reasons for wanting 
the program to continue. For instance, employees have 
experienced many changes in the workplace over the years. They 
stated that the program helps employees to respond when 
changing job requirements demand an increase in math and 
English skills. They feel the skills gained from the program 
help them do their job better and state the hospital benefits 
from better trained employees. 

Personal reasons stated for wanting the program to continue 
are the benefits gained from an increase in self-esteem, a 
sense of accomplishment and the opportunity to continue 
learning . 



Learner Comments 



• "Learn to do what your job needs you to do." 

• "The workforce has changed so much." 

• "It helps the bottom line." 

• "It's good for a lot of people." 

• "Crazy if you don't take it." 
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ii. Education Staff 

Education staff elaborate on their suggestions for expanding 
and improving the program in the next section. 

iii. Managers /Supervisors 

All seven (7) agreed that the pi^ogram should continue, some 
supervisors noted the program fills personal development as 
well as academic skills needs. They stated that even if an 
employee^ does not reach the desired reading goals within the 
time limits of the program, the employee is better off because 
of i;\creased self-esteem. Another said that if employees lack 
basic skills, an opportunity should be made available to them. 
Another supervisor writes: "I want to take this opportunity to 
comment. I feel this is a marvellous opportunity for staff and 
I certainly hope the program continues.'' 

One supervisor stated they did not want the program abused by 
employees, but did not elaborate on this. Another stated that 
we are not coming close to preparing our employees for the 
future . 

iv. Project Team/Discontinued Learners 

The project^ team and discontinued learners give suggestions 
for continuing and expanding the program in the next section. 

V, Summary 

All learners, education staff, supervisors and project team 
clearly state the program is worthwhile and should continue. 
Skills learned assist employees to respond to ongoing changes 
in the workplace and to challenges in their personal lives. 

vi. Conclusions 



• Learners, education staff, supervisors and project team 
want the program to continue for work-related as veil 
as personal reasons. 
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What actions need to be taker ^-Q improve — and — expand the 

program? 

There is evidence that Workplace Eduction has been successful, 
to date, in meeting learner needs. However, for planning 
purposes, learners, education staff, management/supervisors, 
discontinued participants, and the project team were asked for 
program improvement suggestions* In addition, many ideas were 
offered for expanding the program. The responses are discussed 
under course content and teaching style, location, 
scheduling/time, as well as recruitment strategies. 

Course Content and Teaching Style 

Learners generally liked the content and teaching styles. 
They liked the easy-going, friendly, respectful, dedicated, 
fun, unpressured learning environment. Staff indicated that 
learners should continue to get what they "want or ^ need" 
maintaining a sense of program ownership and continuing to 
focus on applied purposes/uses for learning. 

All groups suggested expanding instructional areas. English as 
a second language (ESL) , special help responding to learning 
disabilities, dealing with the issue of clear language and 
additional work-related topics such as memo writing, telephone 
skills, public speaking and computer training are stated. 

In addition, learners and education staff suggest more 
instructor's time be available for learners outside of class 
hours and there should be an opportunity every year to write 
GED at the hospital. Also, some learners would like more one- 
on-one assistance in class with the help of volunteer tutors. 

Education staff note the importance of continuing Angle 
Gillis' role of coordinator, the one-to-one tutoring 
component, and the tradition of respecting employee 
confidentiality. Suggestions include purchasing a greater 
variety of materials, including a computer. Education staff 
and pro j ect team recommend there should be continuity from 
year to year in instructional staff. Furthermore, the project 
team suggests that opportunities be provided "on the job" to 
use and practice new skills gained in the program. 

The staff stated they would like more time and/or funds 
budgeted for professional development. It is beneficial for 
instructors to meet with other workplace educators on a 
regular basis. In addition, the instructor's position within 
the organization requires the same security and benefits .other 
employees receive. This is a necessary step towards increasing 
the status and continuity of the program. 
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ii« Learning Location 



The convenience of on-site location and cost-free access to 
program is, according to both learners and education staff 
one of the program's strengths. The on-site location also 
serve to increase communication and good relationships between 
departments. 

Learners, education staff and project team emphasize the need 
for a permanent classroom and office space. 



iii • Scheduling /Time 

There is a general agreement that more choice of class times 
would accommodate different shifts and departments. The 
instructor needs flexibility to work with all employee 
schedules. Learners have requested that classes be three 
instead of two hours in length and the number of classes 
increased. Some tutors also expressed frustration with 
teaching/learning sessions that are sometimes as little as 
one-half to an hour in length. There is general agreement that 
paid release time is important, although some learners did not 
take advantage of it. One said: "i was on my own time. I 
liked It better that way. It was no strain" on me or my 
employer." The project team indicates that the release time 
policy is sometimes unclear. 

iv. Recruitment and Retention 

Suggestions for recruitment and retention relate to the work 
site, learning environment and distribution of program 
information. 

a. Worksite 

One learner says that concerns about work piling up and 
burdening them when they return from class should be dealt 
with. Education staff agree that learners should not have to 
make-up" work not done or be given less pay. Learners say to 
explain that class is free and there is also a paid release 
policy. They advise orientations for supervisors, head nurses 
and other administrators, letting them know program benefits 
including "better educated staff able to do their job better 
and do new things as well as getting prepared for promotion." 

In addition, education staff state it is necessary to educate 
supervisors about the need for shift; flexibility and to commit 
to matched time. Furthermore, learners should be able' to be 
comfortable approaching supervisors about time off for class 
Learners offered this advice: "Realize that current and 
potential learners might be nervous/shy. Give special 

17 



erJc 



encouragement while at the same time not coercing them." 
Education staff suggest that learners have the support of 
supervisors and co-workers. They recommend better 
communication about time-sharing, expectations and so forth, 
between management/supervisors, instructors and learners. 

Learning Environment 

Learners suggest that recruits are to be assured that in the 
learning situation they can work at their own pace ("Learners 
can do their own thing. Class is not competitive"). Retention 
of learners, to date, has been good. To continueand enhance 
this situation education staff recommend that learners are 
helped to think about their long-term goals, and be provided 
with guidance on how tc put them into practice. Two of the 
three individuals who stopped the program gave "other 
commitments" as the reason. One of the three listed a variety 
of reasons : time of day, felt pressured in math class and 
didn't like the schedule posted in the supervisor's office. 
The one employee who was interviewed for the program but 
didn't register said it was because of "personal things." 
Finally, one instructor says neither instructors nor learn^:. 
should feel guilty if employees find they are unable to 
continue in the program. 

Supervisors suggest to tailor program to individual interests, 
and no pressure to get GED. They suggest to continue the 
program through the summer. One supervisor commented: 
"Learners need to be internally motivated to attend." 

Distributing Program Information 

The project team suggest more publicity and recruitment are 
necessary . The pro j ect team, learners , supervisors and 
educators suggest that learners become more involved in these 
activities. Use former learners as well as department 
managers to talk about benefits they have seen or experienced 
as a result of the program. The education staff recommend 
using as much personal contact as possible to recruit. Word 
of mouth is always a good recruitment method. 

In addition to personal contact, the groups interviewed 
recommended the following activities : develop a workplace 
education handbook, provide awareness workshops , use 
classified section in VINE because- it is read frequently, wear 
the workplace education buttons , have activities during 
education week and a booth on volunteer day . Supervisors 
suggest to handle workplace education "like any other staff 
development activities (e . g. "advertise , people sign up") . 
Education staff suggest providing ongoing education about 
workplace education and "literacy" generally. Furthermore, 
the team and education staff observe that some union 
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relationships need strengthening and that the program staff 
are to be sensitive to union concerns and jurisdictions. 



V. Summary 

Workplace education has been successful, to date, in meeting 
learner needs* To continue to enhance the quality of 
programming for learners and the organization, suggestions 
include more choice of class times and topics . More 
communication and information , particularly for supervisory 
staff, is noted. Recruiting strategies are to be expanded and 
increased. Furthermore, workplace education should have the 
same status as other Training and Development Programs. 

vi . Conclusions 

• The on-site location is convenient and accessible for 
employees, however, a permanent classroom and office 
are required. 

• The no cost policy increases access for employees, 

• Employees like the option of working with tutors as 
well as the "learner-centred" approach which encourages 
input from learners into content, materials, homework 
and so forth. 

• Instructors and tutors can feel a sense of professional 
isolation. They need support within VGH and from 
colleagues in other organizations. 

• Scheduling of classes is important. The time of day 
and/or other priorities may cause some employees to 
discontinue the program. 

The evaluation team set out to gather information on the worth 
of the program and how it could be improved. The findings 
have been summarized into four major questions. Conclusions 
resulted from these summaries. Next, the team shares some of 
their thoughts. The final section lists recommendations which 
follow from the conclusions. 

F. The Project Team Reflects 

Many employees volunteered time to share their personal and 
professional experiences with workplace education. Their 
willingness to provide input is appreciated. The team thanks 
all employees who participated in this evaluation. 

An evaluation of this nature captures much that is important 
in the program, but there is no way of knowing all the many 
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ways employees' lives and work may have been affected or 
changed. W6 are fortunate when employees tell us. 

• A participant of the 90-91 program is near completing 
a degree at Dalhousie University 

• A participant of the 92-93 program has recently enroled 
in the VGH nursing program. 

• Her first public presentation was at the closing 
ceremonies of workplace education. This employee now 
gives presentations on CQI • 

• Her first public presentation at the hospital was 
delivered at the closing ceremonies. She recently wrote 
and delivered a talk on Ward Aide day. 

• Four former learners are enlisted in the tutor training 
and are eager to help other hospital employees who want 
to increase or refresh skills . 

• Although we have no way of knowing exact numbers, many 
employees have successfully written their GED over the 
past three years. 

All of the above continue to be either part or full-time 
employees at the VGH. 

The folD.owing recommendations are intended to maintain and 
enhance the workplace education program - a program dedicated 
to developing those skills and self-confidence that are a 
prerequisite for increased morale, team work and better 
overall performance in the workplace. 

RECOMMENDATIONS 

Introduction 

The following short-term recommendations should be implemented 
prior to starting the 93-94 workplace education program. 

The long-term recommendations will take time to develop and 
implement. However, the project team, coordinator, and 
instructor, should begin to work on them immediately. An 
overall plan v;hich develops goals and a time-line for 
implementing these long-term recommendations should be 
established with the first or second meeting of the project 
team in the 93-94 workplace education term. 

Those recommendations which involve hospital personnel other 
than workplace education staff and pro j ect team are 
suggestions put forth for their consideration. 
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®* Short-Term Recommendations 

Classes are to continue to be held on-site but in a location 
that does not require moving the class from room to room. 

Option A: Get a large room for classes that has a phone 
and space for administrative work and files. 

Option B: Provide an office space and a separate 
classroom* Both are to be dedicated to 
workplace education only. 

Continue: To offer a tutor training course and have 
volunteers work one-on-one with learners. 

To build a library of materials 

To have the coordinator, instructor and tutors 
available to assist learners and supervisors with 
scheduling arrangements and any other needs. 

Increase instructor's hours for one-on-one program-related 
counselling and extra academic assistance. 

Offer greater flexibility in terms of class times to 
accommodate shift and personal demands. Potential learners are 
to continue to have input into most suitable class times. 

Provide the option of more time for each class/ learning 
session. Each class negotiates how much time it needs/has 
available. 

Encourage tutors to help in group math classes as the 
instructor sees the need. 

Give supervisors a minimum of two week's notice of class start 
date and schedules. 

Supervisors should be kept informed about the program in 
general, whether or not they know identities of. individual 
participants. . 

The coordinator and instructor should expand the current 
practices for recruitment, orientation and retention. 
Incorporate suggestions set forth -in this report. 
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Long-Term Reconunendations 



Develop clear guidelines about confidentiality. 

Option A: Supervisors know identity of learners, who are 
able to attend partly on hospital (release 
time) . 

Option B: Supervisors do not know identity of learners 
who then participate on their own time. 

Supervisors should maintain confidentiality but continue to 
support employees . 

The project team should propose a 50/50 release-time policy, 
with the hospital ' s share being either release time or 
equivalent. But consider ceiling for the maximum number of 
hours an employee attends classes . 

Develop a handbook for all levels of employees which outlines 
the purpose and practice of workplace education. 

Develop awareness workshops for supervisors . Within these 
activities , clarify how supervisors should respond if an 
employee approaches them with a basic skill problem. 

Involve supervisors experienced in workplace education as co- 
facilitators in awareness sessions . 

Expand the topics and have more mini-courses. Memo-writing, 
telephone skills, public speaking, and report writing are some 
suggestions. These short sessions will decrease schedule re- 
adjustments on the work-site and reach more people with 
special needs and interests . 

Include purchase of a computer in selection of equipment and 
materials . 

Investigate resources to assist learners with information 
about continuing education. Start with Nova Scotia Community 
College (NSCC) and Employee Assistance Program (CARE) and 
unions . 

Investigate other funding partners. Funding may be sought for 
specific activities in contrast to overall program 
development . 

Continue to search for sources . (time and funding) for 
professional development. 
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Workplace education instructor should be full-time staff 
member with the Department of Training and Development, 

The status of Workplace Education is to be raised to that of 
other training programs in the hospital. 

D. Summary Statement 

The evidence shows that overall, the workplace education 
program has worthwhile goals. The achievement of such goals 
has benefitted individuals and the organization. In addition, 
unanticipated but welcome outcomes have resulted. 
Implementation of the short and long term recommendations will 
ensure that not only will quality programming continue but 
some areas of the program will be improved and expanded. 

A special characteristic of this evaluation is that the team 
has been involved in the entire process. They planned the 
strategy, conducted the interviews and analyzed the data for 
conclusions and recommendations. This unique approach ensures 
that the evaluation questions: "...stem from the information 
needs of relevant and identified decision makers and 
information users." (Patton, 1978, p. 201). Furthermore, the 
skills gained by the employees involved in this process, 
remain within the organization, for application in other- 
areas. 

The project team is to be given credit for their hard work and 
dedication to a program which has touched the lives of many 
and made significant changes in the lives of others over the 
past three years. 

The Victoria General Hospital is to be commended for providing 
such a program* Since implementation of this evaluation, 
increased funding has been approved for 1993-94. 
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APPENDIX: 



Steps in the Process [p. 25] 

Planning Questions [p. 26] 

Data Gathering Questions [p. 27-2 9] 

Data Gathering Plan [p. 30] 

Chart of 92-93 Learner Goals [p. 31-36] 
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STEPS IN THE PROCESS 



Jan, 27 


Christine gives 
presentation on the reasons 
to evaluate workplace 
education, the benefits of 
team approach and 
conunitment requ ired . 
Team consents to proceed. 




March 


2 


Team discusses and analyzes 
concept of evaluation. 




March 


9 


Team discusses why an 
evaluation is useful for 
the program. 




March 


29 


Workshop (Paul & Sue) 
- explored all aspects of 
program and began 
e^aluat ion strategy 



April 5 


Meeting with Learners 
Goals charted 






April 19 


Workshop (Sue and Paul) 
Developed Action Plan 






April 29 


Team discussed plan 






May 3 Team 


divided up tasks of 
data gathering 






August 5 


Workshop (Paul and Sue) 




Analyzed Data for 
conclusion and 






recommendations 






REPORT 
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■ PLANNING QUESTIONS 
Questions for Victoria General Hospital Planning Session on a 
Team Approach toEvaluation 

Please complete this questionnaire as fully as possible. It will 
help us design a workshop to meet your needs. There is no right 
or wrong answer, just your ideas and opinions* 

Name: Position: 



PART I : Background Information 

1. What is your role in the workplace education program? 

2. What kind of assistance do you feel the project team needs 
in developing an evaluation strategy? 

3. Who are the interests groups (stakeholders) involved in the 
workplace education programs? 

4. What are the goals of the workplace education programs? 

5. What sort of evaluation activities have taken place in the 
last three years? 

6. What were you attempting to evaluate? Why? 

?• What did you find out from any evaluation activities that 
were undertaken? 

PART II : Developing an Evaluation strategy 

1. What do you want to find out through an evaluation of the 
workplace education program? (For example, you might want 
to find out if program participants have met their own goals 
or you might want to find out if there has been more 
employee involvement back at the workplace) 

2. Given what you want to find out, what would you see as 
indicators that the program has been successful? (If you 
chose employee involvement , an indicator of success might be 
more program participants on workplace committees) 

3 • Who or v/hat would be sources of information for what you 

want to find out in terms of indicators of success? (If you 
chose employee involvement , you might choose program 
participants as a source of inf orma,tion) 

4. How might you get the information you want? (For example, 
you might interview program participants individually or 
talk to them as a group) 
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